Gen-AI Avatars in Virtual Reality for Job Communication Training

This project focuses on supporting autistic individuals in developing essential job communication skills through an immersive Virtual Reality training system. The application features AI-powered avatars that act as conversational partners, using Large Language Models such as GPT-3.5 Turbo to simulate real-world workplace scenarios. Users interact with these avatars through speech, engaging in customer service conversations and other job-related exchanges.

Furthermore, using a web-based (Node.js) platform, non-tech-savvy users, such as job coaches, can easily create or customize new training situations by entering text prompts, allowing flexible adaptation to individual trainee needs.

Project Team

Ziming Li, Pinaki Prasanna Babar, Mike Barry, Dr. Roshan Peiris

Video

Related Publication

• Ziming Li, Pinaki Prasanna Babar, Mike Barry, and Roshan L Peiris. 2024. Exploring the Use of Large Language Model-Driven Chatbots in Virtual Reality to Train Autistic Individuals in Job Communication Skills. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ‘24). Association for Computing Machinery, New York, NY, USA, Article 156, 1–7. https://doi.org/10.1145/3613905.3651996

• Ziming Li, Pinaki Prasanna Babar, and Roshan L Peiris. 2025. Generative Role-Play Communication Training in Virtual Reality for Autistic Individuals: A Study on Job Coach Experiences in Vocational Training Programs. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ‘25). Association for Computing Machinery, New York, NY, USA, Article 591, 1–22. https://doi.org/10.1145/3706598.3713507