Everyone should be aware of the things that are happening in their home. Yet, for the millions of people living with disabling hearing loss, this still isn’t the case. Recognizing this unmet need, we invented a way to significantly change the day-to-day lives of the Deaf and hard of hearing.

See Sound is the world’s first smart home hearing system for the Deaf. When a noise occurs in the home, See Sound hears and distinguishes the sound, sending a notification to a user’s mobile device identifying the sound. Now, the Deaf can finally see sound.

Even though 5% of the world’s population suffers from hearing loss, the current products on the market are extremely limited. Assistive hearing devices—such as cochlear implants—help indicate or amplify sound, but do not assign meaning to what’s heard. Other visual cues—such as flashing lights—help to notify of a doorbell, incoming phone call, or fire alarm, but are only single use.

There is no product out there that can differentiate sounds. This leaves consumers with a handful of single-sound devices, but nothing that can distinguish a microwave from a baby crying, from glass breaking, and so on.

The limitations of inventing a product like this has been the lack of data. It would take millions of sound samples to create a machine-learning model that could report with any level of confidence. That’s exactly the problem we solved with See Sound, sourcing data from 2 million YouTube clips.

People who are Deaf and hard of hearing lack situational awareness. It’s the innate ability to know what’s going on around you, and that’s largely driven by hearing. It’s easy to take for granted, but not being able to access sounds is disorienting and could mean life or death for the millions of people living with disabling hearing loss.

Deaf people have had to buy various different products to monitor household sounds, but nothing that’s been equipped with sound recognition of this caliber. See Sound is the world’s first home device powered by an AI-learning model that is able to not only listen for a multitude of sounds, but distinguish between them with a high level of accuracy.

This is an idea that not only gives the Deaf community a renewed sense of freedom and control in their home environments, but also makes them safer and more connected than ever.

Conception of Sound Recognition Technology – Hardware Development & Customer Validation

1/2015-12/2015

• Developed prototype of a central, IoT-based, listening device powered by the affordable Raspberry Pi development board

• Gained user feedback to identify Minimal Viable Product features

• Lab & field-tested in multiple locations including in the USA and Chile. Passed with average 72% accuracy rates, proven higher than all algorithms available on the market

Sound Recognition Algorithm – Software Development

1/2015-3/2017

• Completed & finalized sound recognition algorithm & logic development in C programming language

Sound Sensor Hardware Development – In-house Manufacturing

1/2016-3/2017

• Developed in-house microphone-enabled hardware product to optimize sound recognition performance

• Tested several prototypes using different processors to investigate power consumption and efficiency: Atmel, Artik, & Texas Instruments

Sound Model Training – Machine Learning & Artificial Intelligence

10/2016-10/2016

• Researched and developed new methods and approaches to upgraded and improved sound recognition technology using Artificial Intelligence technologies and libraries

• Implemented new version of sound recognition software communicating with AI-trained sound models that improved the maximum sound recognition accuracy threshold at 100%

Patent – US9870719

1/2017

• Title: Apparatus and method for wireless sound recognition to notify users of detected sounds

Patent – US10062304

4/2017

• Patent continuation filing – Pursued additional claims & broadened the scope of the patent

Patient – WO2018195102A1

4/2018

• Patent Cooperation Treaty – Begin patent filing proceedings in EU, India, Canada, South Korea, Japan, Brazil, and China

Meeting with Industrial Designer – Hardware Design

2/2019

• Created customer-facing hardware design to house our required components

Integrated Neural Net – Machine Learning & Artificial Intelligence

1/2019-2/2019

• Utilized deep-learning technology to train semi-supervised model. Achieved benchmark mean average precision

Mobile Application Development

1/2019-Ongoing

• Built app to pair with our device and be the visual system to house the reports for sounds heard

Functioning Prototype Produced

2/2019

• Created first tangible model of See Sound via 3D printing to assess sizing and performance, including sound capture (Refinement ongoing)

Microsite Development

3/2019-4/2019

• Built microsite to showcase the capabilities of the See Sound system

Calibrated Prediction Model

3/2019-Ongoing

• Refined proprietary sound-matching algorithm. Superior performance during internal testing in comparison to machine learning alone

Initiated Alpha Release

3/2019-Ongoing

• Internally distributed See Sound software for testing in various home environments

See Sound is the world’s first smart home hearing system for the Deaf. It’s easy to set up—simply plug the See Sound unit(s) directly into your wall outlet and connect via WiFi to the app on your phone. When a sound occurs, the closest See Sound interprets it and makes a prediction based on its confidence level, alerting the user on their smart devices.

Training a machine-learning model how to distinguish sounds requires millions of sound samples. Those sound samples we needed turned out to be on YouTube in the form of billions of user videos. Working with Google, we were able to leverage a data set of over 2 million human-labeled 10-second sound clips that were manually analyzed, annotated, and organized into the Google Audio Set. Our machine-learning model was then trained with these data to achieve an incredibly high accuracy level.

See Sound’s classification engine has a mean average precision of 0.360, outperforming the state-of-the-art single-level attention model of 0.327 and Google baseline of 0.314. This powers our alpha release to correctly identify over 90% of instances in our 7 certified sounds. We anticipate maintaining this as we finalize our learning phase in the remaining 60+ sounds.

Having earned 3 patents and invested $160,000+ over the last 4 years, we’re ready to launch See Sound worldwide. We are petitioning local, state, and federal US governments to introduce this product under the Americans with Disabilities Act’s (ADA’s) scope of accessible technology, which would allow for the government to cover the cost of the product.

Our pricing model is still proprietary, but we anticipate being similar to Google Home or Amazon Echo. If See Sound was placed in just 5% of Deaf homes in the US (9 million homes), we estimate approximately 450,000 units delivered. If we achieve our goal of being covered under the ADA, we forecast placement in 50% of homes, computing to 4.5M units.

See Sound is poised to drastically change the way Deaf people interact at home, and empower them to lead more independent lives. Finally, a smart device that helps the Deaf see sound.