Audio-Augmented Museum Experiences using Wearable Visual-Inertial Odometry

The auditory sense is an intuitive and immersive channel to experience our surroundings, which motivates us to augment our perception of the real world with digital auditory content. We present a wearable audio augmented reality prototype that tracks the user with six degrees of freedom in a known environment, synthesizes 3D sounds, and plays spatialized audio from arbitrary objects to the user. Our prototype is built using head-mounted visual-inertial odometry, a sound simulation engine on a laptop, and off-the-shelf headphones. We demonstrate an application in a gallery scenario in which visitors can hear objects and scenes drawn in the paintings, feeling audio-visually engaged in the depicted surroundings. In a user study involving 26 participants, we observed that the audio-enhanced exhibition improved people's experience, as well as helped them remember more lively details of the artworks.