The breakthrough came when we managed to shrink YouTube’s captioning model enough to have it run completely on a smartphone.

With a single tap, Live Caption’s machine learning model starts working in real time, completely on-device, no cell data or Wi-Fi required, and the captions never leave your phone.

Every year, mobile experiences and content get more and more creative. But for the 466 million people in the world who are Deaf or Hard of Hearing, that content may be entirely inaccessible. Because, unlike televisions, smartphones do not have built-in captioning for every “channel” or app you are viewing.

That’s why we created Live Caption, a new feature in Android that isolates the audio from any app and automatically generates captions.

Today, captions are expensive and manual to create. For the 466 million people who are Deaf or hard of hearing, that means digital content may be entirely inaccessible. So we set out to make the first phone operating system feature that could caption spoken audio on any content.

The challenge is most automatic captioning models are too large to run on device. We had to invent something new.

Launched successfully on the Pixel 3, 3a and 4 in October 2019, Live Caption is now rolling out to millions of new Samsung and Android devices around the world.

The first time TV broadcast with captions was in 1972. Since then, more and more content isn’t being broadcast on TV at all, it is all on our smartphones. And, today, whether or not your smartphone content has captions is completely up to the creator.

YouTube added automatic captioning in 2009, but what about all the other content that isn’t on YouTube?

So we decided to use the audio feed that is normally only sent to headphones, and build an option that sent it to a speech-to-text model. YouTube’s automatic captioning model required an internet connection, but no one had tried making a version of the model that would work on a smartphone without a data connection.

The breakthrough came when we managed to shrink YouTube’s captioning model enough to have it run completely on a smartphone.

With a single tap, Live Caption’s machine learning model starts working in real time, completely on-device, no cell data or Wi-Fi required, and the captions never leave your phone.

Today, Live Caption is a built-in feature on Pixel 5, 4, 4a, 4XL, 3, 3a, and 3 XL. Samsung has added Live Caption on their new phones. Over the next 12 months we are working with more Android phone manufacturers to build Live Caption into their phones (with a goal of ultimately reaching all 2 billion Android users).

We also have plans to support more languages and continue to improve the accuracy of the captions over time.