With the rise of on-demand TV shows and movies, viewers have a dizzying array of entertainment options to explore. Cable provider Comcast has been helping customers navigate this expansive content landscape using AI via its Xfinity voice remote.

The remote taps machine learning to help customers decide what to watch and when to watch it, providing users with a tailored at-home video experience, Comcast CTO Matthew Zelesko explained at VentureBeat’s virtual Transform 2021 conference.

“The content landscape has grown dramatically. And we realized that it was much harder for customers to determine even simple questions, like what to watch and where to watch it,” Zelesko said. “Navigating to the content you want is really so important. And we felt voice was just the most natural way to do that.”

Comcast invested in AI to find better ways to provide a personalized viewing experience, Zelesko said. Doing so required a deep understanding of content and how to provide it, and the voice remote helped streamline this process. Comcast released the first version of the remote as an app in 2015 to explore voice interfaces. The company said it received overwhelmingly positive feedback and shipped millions of remotes in the first year.

“More than 70% of our customers with a voice remote use voice commands at least once a week. And those who use the remote consistently spend more time watching more content than those who don’t,” Zelesko said. “So when we make it easy to discover what they want, customers use more of our services, and they use them longer.”

AI backed by data

Comcast attributes the remote’s success to the sheer amount of data it consolidates, which helps it understand users’ preferences. Understanding content metadata, like genre and cast, is critical to the customer’s content discovery experience. To organize its knowledge, Comcast uses CoMPASS, a cloud-based metadata platform to make informed decisions about consumer preferences and experiences.

A new feature called deep metadata utilizes computer vision to understand what is happening in scenes, which could allow customers to jump to the most important highlights in a recorded football game, for example. It uses the show’s closed captioning and computer vision of the video to pinpoint specific on-screen highlights to feature.

To ensure accurate labeling, the metadata classification relies on overlapping annotations — where a small percentage of images analyzed by computer vision are also annotated by two human analysts and assessed for discrepancies to refine the algorithm.

“Without that great organized metadata about all of the hundreds of thousands of pieces of content, no amount of AI or machine learning is going to get customers to what they want,” Zelesko said.

Buy or build conundrum

Comcast licenses much of the technology it builds internally, like the voice remote, to other cable providers in the United States, Canada, and Europe. Zelesko said Comcast reviewed available commercial products before developing the remote and realizing nothing on the market met its need for accurate and easy-to-use natural language processing. No other company was indexing the quantity and complexity of metadata, so “no off-the-shelf platform was going to work,” Zelesko said.

“We still buy where it makes sense, and where there’s a great solution that meets our needs in the market. But given our scale, and also really the unique nature of the problems that we’re solving, we find that we get the best results when we build to meet our customers’ needs,” he said.