  The paper introduces AniTalker, an innovative framework designed to generate
lifelike talking faces from a single portrait. Unlike existing models that
primarily focus on verbal cues such as lip synchronization and fail to capture
the complex dynamics of facial expressions and nonverbal cues, AniTalker
employs a universal motion representation. This innovative representation
effectively captures a wide range of facial dynamics, including subtle
expressions and head movements. AniTalker enhances motion depiction through two
self-supervised learning strategies: the first involves reconstructing target
video frames from source frames within the same identity to learn subtle motion
representations, and the second develops an identity encoder using metric
learning while actively minimizing mutual information between the identity and
motion encoders. This approach ensures that the motion representation is
dynamic and devoid of identity-specific details, significantly reducing the
need for labeled data. Additionally, the integration of a diffusion model with
a variance adapter allows for the generation of diverse and controllable facial
animations. This method not only demonstrates AniTalker's capability to create
detailed and realistic facial movements but also underscores its potential in
crafting dynamic avatars for real-world applications. Synthetic results can be
viewed at https://github.com/X-LANCE/AniTalker.
