Revolutionizing Video Creation: Adobe Unveils Groundbreaking Firefly Video Models

Earlier this year, Adobe shared some of its research into generative AI for video, with initial tests showing it removing unwanted elements from a shot, filling gaps in the timeline with generated footage, and other “magic” tricks. The new Adobe Firefly video model is expected to join the company’s generative AI family in the second half of 2024.

AI Image GeneratorAdobe FireflyFirefly has become quite popular among creatives because, firstly, it uses training data that developers have legal rights to, making it more commercially safe and ethical than similar AI tools from competitors, and secondly, many of Firefly’s features are directly integrated into Adobe software, such as AI vector generation in Illustrator and Generative Fill in Photoshop.

What Adobe has so far failed to do is release Firefly for video.

Adobe Firefly Video Models

During this year’s NAB preview, Adobe showed off a number of AI tools for filmmakers and video creators, including text-to-video and image-to-video generators, as well as handy editing features like adding objects, masking distracting elements with one click, and extending shots directly in the timeline. Check out the full demo session here.

Adobe announced that the beta version of the Firefly Video Model is ready and ready to go to market. According to an Adobe press release, the first AI-powered features for video will include at least what they call generative extend, and the ability to create videos from text and image prompts.

For example, you can use precise control like camera angles and camera motion to create animations and effects, use camera motion to generate inserts into key video shots, or combine your video with professionally shot footage to tell your story faster.

From Adobe’s press release:

Future Schedule

But there’s still no official launch date: As Adobe developers said in a press release, getting the AI-powered tools out quickly isn’t as important to the company as getting it right.

According to the announcement, the new Firefly Video Model will be available “later this year.” Once launched, the text-to-video and image-to-video generators will beFirefly.Adobe.comYou can test it out in the Preview section, and Generative Extend is embedded directly into Premiere Pro. It’s not clear yet when other preview features (such as object removal) will be available in production.

When Firefly Video Model becomes available as a beta version,HereYou can join the list to receive the latest updates.

AI dubbing and lip sync

Apart from the features already demoed, Adobe also announced several new tools powered by Firefly for enterprise services, the most impressive of which is AI dubbing and lip sync, which allows users to translate spoken words in a video into different languages while ensuring lip sync with the original speaker’s voice.

Images source: Adobe

The service is already available to enterprise customers as a beta API for seamless integration into existing workflows, and as a web application. We haven’t tried it yet, but the demo presentation at Adobe’s live event was shockingly convincing.

Feature image source: Adobe.