Enhancing Real-Time Accessibility Through Vibration Feedback

Captions make audio content accessible for Deaf and Hard-of-Hearing (DHH) individuals, yet they can fall short in real-time, multi-speaker settings where identifying who is speaking becomes difficult. To address this challenge, we developed Haptic-Captioning, a system that translates sound features into vibration feedback on the wrist, offering an additional sensory channel to complement visual captions.

Project Team

Yiwen Wang, Ziming Li, Pratheep Kumar Chelladurai, Dr. Wendy Dannels, Dr. Tae Oh, Dr. Roshan Peiris

Related Publication

• Yiwen Wang, Ziming Li, Pratheep Kumar Chelladurai, Wendy Dannels, Tae Oh, and Roshan L Peiris. 2023. Haptic-Captioning: Using Audio-Haptic Interfaces to Enhance Speaker Indication in Real-Time Captions for Deaf and Hard-of-Hearing Viewers. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ‘23). Association for Computing Machinery, New York, NY, USA, Article 781, 1–14. https://doi.org/10.1145/3544548.3581076