The world’s roughly 200 million visually impaired people have been excluded from the excitement of live sporting events. Visual impairment makes it difficult to follow a game and to enjoy the action live with others. As we discovered, this stems from the information gap with regular spectators. Even as more people with disabilities participate in sports through the Paralympics and other events, accessibility in sports viewing is still lacking. Envisioning a barrier-free future where people of all abilities can enjoy the same live sports events, we launched this project to assist the visually impaired in enjoying motorsport.

From seeing the game, to hearing the game. We developed Voice Watch, the world’s first play-by-play commentary AI, to enable visually impaired people to follow live races. The technology converts various kinds of race data into audio, providing commentary for visually impaired spectators at circuits without live commentary. Voice Watch automatically generates live commentary to create a new motorsport spectator experience: “hearing” the race.

Voice Watch consists of three different AIs: an object recognition AI, a sign detection AI, and a speech frame AI. The object recognition AI uses deep-learning on Google Images results featuring cars from each team in the race. It serves as the visually impaired person’s eyes, using cameras to distinguish the cars and grasp the race unfolding before them. The sign detection AI analyzes real-time driving data to identify signs of major changes in the race, such as a change in positions. Based on patterns in the way races develop, the AI predicts what happens next. For the speech frame AI we analyzed 50 hours of play-by-play audio from Japan’s most renowned professional race commentator, distilling this expertise into unique speech frames capable of reproducing the commentary phrases that excite spectators. By bringing these elements together, Voice Watch generates live commentary that enables visually impaired spectators to enjoy the races.

We introduced Voice Watch into Japan’s leading endurance racing series, Super Taikyu, during events held at the Suzuka Circuit and Okayama International Circuit, both of which host F1 races. Using an object identification model trained on images of team cars, and iPhones among the spectator seats as network cameras, Voice Watch analyzed the unfolding scenes in real-time. It also gathered numerical information from the driving data monitor available on the event website, which is updated with each team’s times and positions. By recording all stats throughout the race, comparing them over time, and calculating the differences between teams, Voice Watch forecasted over 150 potential future outcomes, such as overtaking and changes in the standings and included them in the commentary. The result was a highly accurate live commentary that allowed visually impaired spectators to enjoy races at venues without live commentary.

Of the visually impaired individuals who tried Voice Watch, 80% said they were able to enjoy the race and would like to attend other sporting events if Voice Watch was available. These results indicate that we were successful in creating a new spectator experience for the visually impaired. Media exposure reached over 2 million. The positive response led to Voice Watch also being adopted at Kodomo Suporinpikku, a children’s sporting event in Tokyo attended by some 1,000 people. In the 50m sprint for elementary school children, a visually impaired father was able to enjoy his child’s run via Voice Watch’s commentary. Many other sporting events, from minor sports tournaments to regional qualifiers, have a passionate following but lack live commentary. In the future, we aim to move beyond motorsport to create a world where everyone can enjoy all kinds of live sporting events, regardless of physical ability.