  Vivid talking face generation holds immense potential applications across
diverse multimedia domains, such as film and game production. While existing
methods accurately synchronize lip movements with input audio, they typically
ignore crucial alignments between emotion and facial cues, which include
expression, gaze, and head pose. These alignments are indispensable for
synthesizing realistic videos. To address these issues, we propose a two-stage
audio-driven talking face generation framework that employs 3D facial landmarks
as intermediate variables. This framework achieves collaborative alignment of
expression, gaze, and pose with emotions through self-supervised learning.
Specifically, we decompose this task into two key steps, namely
speech-to-landmarks synthesis and landmarks-to-face generation. The first step
focuses on simultaneously synthesizing emotionally aligned facial cues,
including normalized landmarks that represent expressions, gaze, and head pose.
These cues are subsequently reassembled into relocated facial landmarks. In the
second step, these relocated landmarks are mapped to latent key points using
self-supervised learning and then input into a pretrained model to create
high-quality face images. Extensive experiments on the MEAD dataset demonstrate
that our model significantly advances the state-of-the-art performance in both
visual quality and emotional alignment.
