Seeing AI is a free app that narrates the visual world for the blind and low vision community. This ongoing research project harnesses the power of Artificial Intelligence to open up the visual world and describe nearby people, text, currency, color and objects with spoken audio. Much like a Swiss Army knife, this app provides many tools in one by leveraging the power of on-device deep learning to open many “first time in life” scenarios. Key features include - Real time text reading, document structure understanding, audio-based barcode locator and product recognizer, face recognition and emotion/age/gender description, currency recognition, color recognition, audible light detector and handwriting reader. Additionally, this app can make even other apps accessible and inclusive by describing photos while using other apps. See the 90 second product video : http://aka.ms/seeingaiproductvideo

With a uniquely creative audio-based user experience, Seeing AI runs on-device AI to solve problems faced by the blind community in a manner that had never been done before. Given that blind users often have difficulty framing camera correctly, the app uses realtime AI on-device to guide them to take a better photograph resulting in higher accuracy - e.g. audibly guiding till all 4 corners of a document are visible.

Similarly, to be able to recognize barcodes, the app generated audio beeps to help guide the user towards areas with barcodes. Previously, blind users would have to buy a bulky $1300 dedicated hardware barcode scanner to be able to do this.

When the camera sees a face in front in the person mode, it describes how far the face is, its location in the picture to help users frame it, and if the face is smiling or another emotion. With all this audible feedback, blind users are already posting smiling photographs of their friends on social media. Additionally, users can take three photos and train an face recognizer (a first in the industry), so if a friend appears in front of the camera, it will announce it in real time.

Since launch in July 2017, the app has improved independence by assisting users in completing over 7 million tasks independently and has been downloaded by 200,000 users as of May 2018. Many of these tasks previously required relying on sighted assistance.

Rated 4.8 stars on the Apple App Store, feedback often contains words such as ‘new confidence’, ‘new opportunities’, ‘new experiences’, ‘game changer’, ‘’independence’ and ‘empowerment’. A subset of feedback received often revolves around the following themes :

* First time experiences in life – reading menus, selfies, finding items in a vending machine

* Education – Blind students in school can now read inaccessible paper text. Similarly, blind parents are reading books, checking the handwritten homework of their kids and notes from teachers.

* Employment

* Feeling included in social conversations

* Blind photography

* Grocery shopping and self-reliance in the kitchen

* Getting access to affordable technology.

Our mission is to empower every person on the planet to achieve more. We recognize that the blind and low vision community (estimated size of 253 million according to WHO) is often underserved by technology and historically has reduced educational and employment opportunities. And we want to close that gap with the help of technology. Starting as a grassroots innovation project, the team built artificial intelligence (like describing images in english sentences) and mixing with a creative audio interface, and releasing this as a free app to get it in the hands of as many users as possible.

Feb 2014 - First prototype of running Deep learning network on Phone

July 2015 - A group of employees with different skills meet at a week long programming event to develop solutions for the blind community. After the event, they keep progressing in their passion time, refining the technologies and making them faster.

Jan 2016 - A team of interns help build first prototype of app

May 2016 - Team of 3 developers gets official funding, start building the app.

July 2017 - After one year of development, app releases silently. Knowledge of the app spreads through word of mouth in the blind community, making it the most searched app on Apple’s App Store. Apple features it on the App Store homepage.

Dec 2017 - Version 2 of the app launches.

During initial phases of the project, we discovered three key issues with existing accessibility tools:

Lack of access to technology due to expensive tools. For example, most smartglass solutions for the blind are in the range of $2000-$10000, even with limited functionality. Similarly, hardware based laser barcode scanners designed for the blind community cost upwards of $1300. For a community which historically has reduced educational and employment opportunities, such costs make it even more harder to get access to these technologies.

Limited functionality : Even with these costs, many of the hardware based solutions have limited or single purpose functionality

Bulky hardware: With hardware solutions, its hard to carry around bulky tools. Since many of these hardware tools do one feature, this means carrying multiple items in your bag, reducing mobility.

With Seeing AI App, we hope to make the physical world more accessible, with affordable technology.