  Generating talking head videos through a face image and a piece of speech
audio still contains many challenges. ie, unnatural head movement, distorted
expression, and identity modification. We argue that these issues are mainly
because of learning from the coupled 2D motion fields. On the other hand,
explicitly using 3D information also suffers problems of stiff expression and
incoherent video. We present SadTalker, which generates 3D motion coefficients
(head pose, expression) of the 3DMM from audio and implicitly modulates a novel
3D-aware face render for talking head generation. To learn the realistic motion
coefficients, we explicitly model the connections between audio and different
types of motion coefficients individually. Precisely, we present ExpNet to
learn the accurate facial expression from audio by distilling both coefficients
and 3D-rendered faces. As for the head pose, we design PoseVAE via a
conditional VAE to synthesize head motion in different styles. Finally, the
generated 3D motion coefficients are mapped to the unsupervised 3D keypoints
space of the proposed face render, and synthesize the final video. We conducted
extensive experiments to demonstrate the superiority of our method in terms of
motion and video quality.
