Visualizing Sound for Deaf and Hard-of-Hearing Users in Virtual Reality

Project Team

Ziming Li, Shannon Connell, Dr. Wendy Dannels, Dr. Kristen Shinohara, Dr. Roshan Peiris

SoundVizVR System

This project explores how visual cues can make sound-based information in VR more accessible to Deaf and Hard-of-Hearing (DHH) users. In typical VR experiences, sounds carry key details about space, interaction, and immersion. However, when sound is inaccessible, users may lose important contextual information.

SoundVizVR introduces a visual system that represents sound properties and types directly in the virtual environment. The system features two components:

• Sound-Characteristic Indicators, which visualize a sound’s loudness, duration, and spatial location through techniques such as On-Object Indicators and Mini-Maps.

• Sound-Type Indicators, which add contextual information about what kind of sound is occurring, using icons and text.

Source Code: Github Repository

Video

Related Publication

• Li, Ziming and Peiris, Roshan (2021) “VR Sound Mapping: Make Sound Accessible for DHH People in Virtual Reality Environments,” Frameless: Vol. 4: Iss. 1, Article 10. Available at: https://repository.rit.edu/frameless/vol4/iss1/10

• Ziming Li, Shannon Connell, Wendy Dannels, and Roshan Peiris. 2022. SoundVizVR: Sound Indicators for Accessible Sounds in Virtual Reality for Deaf or Hard-of-Hearing Users. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ‘22). Association for Computing Machinery, New York, NY, USA, Article 5, 1–13. https://doi.org/10.1145/3517428.3544817

SoundVizVR As A Plugin for Game Engine

Building upon the SoundVizVR system for visualizing sounds in virtual reality, this project introduces the SoundVizVR Plugin, a Unity-based tool that allows VR developers to easily integrate the SoundVizVR visualization system into their own applications and games. The goal was to bring sound accessibility into the hands of developers and explore how such tools can shape inclusive design practices in VR.

Plugin Download (Unity Package): Github Release

Source Code: Github Repository

Video

Related Publication

• Ziming Li, Kristen Shinohara, and Roshan L Peiris. 2023. Exploring the Use of the SoundVizVR Plugin with Game Developers in the Development of Sound-Accessible Virtual Reality Games. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA ‘23). Association for Computing Machinery, New York, NY, USA, Article 130, 1–7. https://doi.org/10.1145/3544549.3585750

• Ziming Li and Roshan Peiris. 2024. Exploring the SoundVizVR Plugin in the Development of Sound-Accessible Virtual Reality Games: Insights from Game Developers and Players. ACM Trans. Access. Comput. 17, 4, Article 18 (December 2024), 20 pages. https://doi.org/10.1145/3698882