Deepdub GO uses generative AI platform to dub any video content professionally

AI

Language localization startup Deepdub Ltd. today announced the launch of a new, generative artificial intelligence-powered dubbing tool that anyone can use to dub video content into 65 different languages.

The new tool is called Deepdub GO, and it uses generative AI technology similar to that which powers OpenAI LP’s famous ChatGPT, albeit for a very different use case. It’s a more accessible version of of Deepdub’s proprietary AI platform for Hollywood studios, which is used to dub TV shows and movies in different languages.

As Deepdub explains, dubbing has been used by the entertainment industry for years to create a more engaging and immersive experience for viewers who don’t like to rely on subtitles. It makes content more accessible to wider audiences, but the complexity and high cost of the process means that dubbing has generally been used only by film studios and video streaming platforms such as Netflix Inc.

Deepdub GO makes dubbing available to everyone, founder and Chief Executive Ofir Krakowski told SiliconANGLE. He explained that he and his brother and co-founder Nir spent months exploring different potential use cases for generative AI, looking for ways in which the technology could make a positive impact.

“We realized that content is always created in a specific language and culture, which makes it inaccessible to billions of people around the globe,” Krakowski said. “As we are both huge fans of the magic of great storytelling and content creation, we realized dubbing was a perfect fit. It’s not just stories, but most human knowledge that’s not crossing the language barrier.”

So Deepdub’s mission is to use generative AI to make content and knowledge more widely available through dubbing. To dub any kind of content, users simply upload the video to the Deepdub GO platform, select the video style and desired languages, and the platform does the rest. Once the initial dub is completed, users can then edit and adjust it with various tools to ensure it sounds just right.

“Deepdub GO is a web-based professional platform that elegantly encapsulates much of the same state-of-the-art generative AI technology that we use for movie and TV dubbing,” Krakowski explained. “Its user-friendly interface enables experts and newcomers alike to create and localize audio-visual content with vivid and natural voices in dozens of languages. As the platform was initially designed for Hollywood productions, it incorporates best-in-class security measures.”

Krakowski said Deepdub GO enables users to translate and dub any kind of content, including fitness videos, educational content or even TikTok posts. Features include realistic, humanlike voices, emotion-prompting and voice-guiding technology that enable users to edit the dubbed voices.

The emotion-prompting feature is especially clever, giving editors fine-grained control over the emotional expressivity of the dubbed voice. According to Krakowski, it uses a customized large language model to steer the performance of the voice through text-based prompts.

“It’s based on text and language understanding,” he explained. “The basic voice style that will be created will have the emotion associated with this kind of text — similar to how a human would say it. You can mark any part of the text and enhance the emotion or change it to a different style of emotion that better suits your needs, such as giggly, whispering, shouting and laughing.”

Meanwhile, voice-guiding is a technique that makes it possible to achieve theatrical-grade audio quality. Users provide a voice recording to act as a guide for the emotional expressivity of the dubbed audio, and Deepdub GO will replicate it in the target language and voice, Krakowski said.

The platform also supports voice cloning, meaning it’s possible to translate the original voice into foreign languages such as Portuguese, Spanish, German, French or Japanese. Otherwise, users can choose from thousands of artificial voices. The extensive editing capabilities also make it possible to alter the translations themselves, voice characteristics, the duration of the translated speech and even the word count to achieve more accurate lip-synchronization.

Constellation Research Inc. analyst Andy Thurai said dubbing is one of the coolest and most useful applications of generative AI as it enables video content to be translated with minimal effort. It can be applied to TV commercials, online learning, virtual human presenting, real-time translation of executives and politicians’ speeches, he said.

“Most enterprises have shied away from localizing video and audio content until now because it is very expensive to do,” Thurai said. “DeepDub is very effective. In the case of an audio rewrite, it takes the complete script and rewrites it in a different language and it also takes care of the additional complexity of lip syncing. This can also help with live bleeping of words, or masking, changing it to something else on the fly, and live translation as it happens.”

Thurai noted that DeepDub Go’s emotion prompting and emotional resonance capabilities can help to reduce the likelihood of machine-like voices that lack emotion, which has been a major problem with dubbing software. However, he pointed out that DeepDub is not the only AI-powered dubbing platform in town.

“There are a good many alternatives out there,” he said. “One of my favorites is a company called Flawless AI, which offers a very capable solution of its own. In addition, Google Translate offers similar capabilities for free. DeepDub Go’s major differentiator is its support for 65 languages, which can be compelling for many.”

Deepdub GO is available now as a standalone platform, and also as an application programming interface, so it can be integrated into existing video editing tools and workflows. It’s launching with a free plan, a basic plan and a professional plan to accommodate the needs of different users.

“Deepdub GO is an enabler of global storytelling, allowing content creators to resonate with their audiences in a deeper, more personal way,” Krakowski added. “With our studio platform, creators can amplify the potential reach of every piece of content beyond traditional geographical and linguistic boundaries.”

Image: Deepdub