Virtual Reality and Artificial Intelligence in Auditory Perception

Project Acronym: VRAI
Duration: 01/07/2024 – 30/06/2028
Project Number: APVV-23-0054

Abstract

In everyday situations, humans are exposed to multiple concurrent stimuli in complex, continuously changing environments. They often need to determine the sound source location or focus on one speaker in the mixture of other sounds. To correctly extract relevant information, humans adapt their processing to reflect the specifics of the current scene, and they learn from previous experience to improve the perceptual strategies used. Existing tools that assess and enhance hearing abilities do not adequately capture the rich and inherently motivating real-world auditory scenes. The current proposal aims to use Virtual Reality (VR) and Artificial Intelligence (AI) to develop such tools for spatial hearing tasks and to enhance our understanding of the representation and adaptation of spatial processing. Specifically, virtualization (VR) and gamification (AI) are promising paths to increase ecological realism in auditory training and testing. VR + gamified tools can immerse the participants in rich audiovisual environments, thus effectively addressing the insufficient content of existing lab-based methods. The current project aims to use VR+AI tools to examine the mechanisms of adaptation and representation in 1) distance and 2) horizontal localization, and to develop training tools for improving spatial hearing in terms of 3) horizontal localization and 4) speech intelligibility in complex listening environments. Also, advanced AI-based computational tools will be used to analyze neural imaging data, enhancing our understanding of the neural mechanisms of spatial auditory processing. In addition to normal-hearing listeners, the results and training tools will be usable for hearing-aid and cochlear implants wearers as they need to adapt to new neural encoding of space, different from the “normal” one. More generally, the results can be applied in telecommunications, VR and augmented reality, robotics, telemedicine, and other fields where communication occurs.