MIMS Final Project 2019

Augmented Reality for Visually Impaired People

About 36 million people worldwide are blind. For many of these people, navigating new spaces can be a cumbersome or frustrating experience as they listen and feel their way around their environment.

Our project seeks to utilize the spatial mapping power of the Microsoft HoloLens augmented reality device in order to help users navigate in unfamiliar environments.  This could improve independence for blind people by allowing them to find and read objects such as bus stops without sighted assistance.

By making audible the 3D mesh of the user’s surroundings that these devices generate, we will create a soundscape that will let users hear the objects around them. Thus, visually impaired users will be able to hear the presence of obstacles like tables and chairs, or the lack of obstacles that represents a doorway, from up to five meters away.

In combining this capability with machine vision-powered text recognition in one hands-free headset, we hope to give blind users a new level of independence in unfamiliar spaces.

Source code available via Github: https://github.com/arvips/Mesh-Manipulation 

Augmented Reality for Visually Impaired People icon. Created by Lloyd Humphreys from Noun Project.
Augmented Reality for Visually Impaired People icon. Created by Lloyd Humphreys from Noun Project.
A blind test user wearing an AR device points towards a bus stop
AR for VIPs Blind User Testing
A pool table with a mesh overlay and several red balls representing audio beacons
Objects are mapped and assigned audio sources so users can hear them at range
AR for VIPs Poster
AR for VIPs Poster

Video

AR for VIPs Demonstration Video

AR for VIPs Demonstration Video

Last updated:

May 16, 2019