Bring Anything to Life: X-Portrait 2 Revolutionizes Animation with Jaw-Dropping Realism
Last week, Runway launched a generative character performance tool that can convert videos into any style of virtual character animation while maintaining expression, voice, and mouth synchronization. Just use a camera to record the actor’s performance, and Act-One can convert the video into an animation of the virtual character. For example, the actor’s eyes, facial expressions, movement rhythm, and speaking style will all be captured.
This week someone from ByteDance came to me and told me that they also have a similar product in internal testing, which is even better than Runway’s Act-One, and asked me to test it.
ReallyI don’t know if something unexpected happens, I’ll be shocked if I test it. There are indeed many good things in Byte, but they are all hidden.
There is currently no official product name for this tool internally, it is calledX-Portrait 2。Just by looking at the name, you can tell that I have been researching it for a while, and it is already 2 generations old.
X-Portrait 2 is an efficient portrait animation generation tool based on deep learning. Users only need to providea static portraitand a“Driven videos” with expressions and movementsX-Portrait 2 can convert theExpressions and actions transferred to static imagesgenerate natural, smooth and expressive animations.
It can not only transfer the action expressions of people in the video to the target image, but also capture and restore extremely subtle changes in facial expressions, such asPouting, bulging cheeks, frowningetc., so that the animation generated by the transfer is not only smooth, but also conveys rich emotions.
Not much to say, let’s take a look at a few cases I tested.
X-Portrait 2 can accurately capture and transmit rapid head movements, and can even restore the subtle expression changes and emotional changes of the characters in the video. Make the generated animation appear more realistic and vivid.
Xiang Zuo also has acting skills
The model is highly adaptable and can achieve cross-style expression transfer between different styles, such as real portraits and cartoon images.
It is suitable for both real-person portraits and virtual images such as cartoons and comic characters.
In the past, this required actors to wear motion capture equipment or use camera motion capture technology to complete, but now it only requires simple pictures and videos to use prompt words to control.
Separation of “face” and “expression”: only the expression changes, not the face
In order to prevent the photo from losing its original appearance when moving, X-Portrait 2 adopts the method of separating “face” and “expression”. This method is like separating a person’s appearance and expression, allowing only the expression to change without changing the original facial features.
This separation method allows the photo to always maintain its original appearance when imitating the video expression, for example, the face shape will not be affected by the expression.
Fine motion restoration: capture every detail
X-Portrait 2 is very sensitive to small expressions and fast movements. For example, a quick head turn, a pouted mouth or a slight raising of eyebrows, these details will be captured and restored by the model, and the resulting video effect is very delicate. This fine motion restoration makes it particularly suitable for film and television special effects or animation production, making the generated characters look more realistic.
Compared to state-of-the-art methods such as X-Portrait and the recently released Runyway Act-One, X-Portrait 2 can faithfully represent fast head movements, subtle expression changes, and strong personal emotions, which are essential for high-quality content creation. (such as animation and film production) are crucial.
Technical innovation points:
1. High-precision expression encoder: achieve true reproduction of subtle expressions
- Capture subtle emotional changes: X-Portrait 2’s expression encoder is trained on large-scale data sets to capture and restore complex facial details and emotional changes. For example, it can accurately reproduce small but key expressions such as pouting, bulging cheeks, and frowning, which makes the generated animation not just a mechanical imitation of expressions, but full of personality and delicate emotions.
- High-fidelity expression transfer: This encoder retains the emotion and tone of the original video during the generation process, making the generated expressions more natural and accurately conveying emotional intensity, providing creators with an animation generation experience that goes beyond traditional methods.
2. Strong separation between appearance and motion (Appearance and Motion Disentanglement)
- Separate appearance and expression changes: The technical architecture of X-Portrait 2 separates the appearance of the image from the expression movements, allowing the model to focus only on the transfer of expression and movement information without changing the appearance of the static portrait. This separation ensures the independence and consistency of expression generation, making expression migration more natural especially when dealing with complex dynamic changes.
- Support multiple styles of applications: The separation of appearance and action also means that the model can be easily applied to different styles of images. Whether it’s a realistic portrait or a cartoon character, X-Portrait 2 can accurately transfer expressions to the target style. This cross-style capability enables creators to integrate image materials of different styles into one project, enriching the expressiveness of creation.
3. Innovative applications of generative diffusion models
- Multi-view training and diffusion generation: Using a generative diffusion model, trained on multi-view data. This model can restore the changes in expressions under different viewing angles, making the animation generation effect more smooth and realistic. Through multi-view training, the diffusion model can ensure that facial expressions are naturally coherent at every angle, avoiding the incongruity problem of traditional methods when angles change.
- Denoising mechanism and consistency optimization: The diffusion model uses a denoising mechanism during the generation process to make the generated images higher quality and reduce the noise generated in expression and action transformations. This denoising process ensures clarity of complex expressions and fast movements, resulting in smoother, more refined animations.
4. Highly adaptive cross-domain expression transfer capabilities
- Support cross-domain applications: X-Portrait 2’s cross-domain migration capability makes it suitable for animation needs in different styles and fields, and can easily migrate expressions from real portraits to virtual characters, comic-style and other styles. This cross-domain adaptability allows the model to be used flexibly in creation, providing creators with a wider range of style choices.
- Multiple driver input compatibility: Supports the use of multiple types of driver videos, either movie footage, animation or user-recorded video. This compatibility not only improves the applicability of the tool, but also provides creators with greater freedom in selecting driver videos, allowing them to choose the most suitable driver source for different needs.
5. Improvement of realism and dynamic expression
- Realistic performance and detail capture: It can meticulously restore the character’s rapid head movements, subtle facial changes and emotional characteristics, improving the realism of the generated animation. Compared with traditional methods, this model has obvious advantages in high dynamic expression, making the generated animation closer to the effect of real images.
- Movie-level animation quality: Excellent performance in generating dynamic scenes and can be used in high-quality film and animation production. Whether it is subtle emotional transmission or dramatic expression changes, X-Portrait 2 is able to maintain coherent expression fluency, bringing movie-level animation quality to content creation.