A new software called Lipdub has been released by Captions, an AI-powered video editing business, to translate clips into 28 different languages.
French, Hindi, Spanish, Italian, Portuguese, Japanese, and other languages are supported by Lipdub, which is free to download from the App Store. Users of the app may even translate films into baby babble, Gen Z, pirate, and Texas lingo. The demonstration video demonstrates how the software can alter lip movement following the chosen target language. In any case, there is occasionally a slight delay between the audio and the lip movement.
Users can interpret and post videos of one person speaking for up to one minute on other social networking sites.
According to Captions’ website, the self-titled video editing program has been used by more than 3 million creators. According to the startup, it has more than 100,000 daily users. The Captions app has several AI-powered video editing tools, including removing “ums” and “ahs,” lowering background noise, and improving pronunciation. Additionally, the app contains an “AI Lipdub” tool that can alter lip movement during post-production editing if the transcript is changed.
Gaurav Misra, Snap’s head of design engineering, established Captions in 2021. The business received $25 million in a Series B investment in June that was co-led by Kleiner Perkins and included Sequoia Capital, Andreessen Horowitz (a16z), and SV Angel. Funding for Captions has totaled $40 million to date.
A rising trend is using AI dubbing and translation to reach more people. YouTube said in June that it is testing a machine learning-based solution that will enable users to subtitle their videos in foreign languages automatically. According to the business, even better lip-syncing is being worked on. The company said last month that it is integrating AI-powered dubbing right into YouTube Studio for customers wishing to translate videos into other languages for more straightforward access.
ElevenLabs’ dubbing tool, which supports 29 different languages, was unveiled earlier this month by the AI-powered voice-generating platform. As previously reported by Rest of the World, organizations that provide dubbing services make millions of dollars by translating videos for well-known YouTubers like MrBeast.
Startups that use AI for dubbing have attracted much investor attention, with companies like Papercup in the UK and Deepdub in Israel raising millions of dollars.
