  While recent research has made significant progress in speech-driven talking
face generation, the quality of the generated video still lags behind that of
real recordings. One reason for this is the use of handcrafted intermediate
representations like facial landmarks and 3DMM coefficients, which are designed
based on human knowledge and are insufficient to precisely describe facial
movements. Additionally, these methods require an external pretrained model for
extracting these representations, whose performance sets an upper bound on
talking face generation. To address these limitations, we propose a novel
method called DAE-Talker that leverages data-driven latent representations
obtained from a diffusion autoencoder (DAE). DAE contains an image encoder that
encodes an image into a latent vector and a DDIM image decoder that
reconstructs the image from it. We train our DAE on talking face video frames
and then extract their latent representations as the training target for a
Conformer-based speech2latent model. This allows DAE-Talker to synthesize full
video frames and produce natural head movements that align with the content of
speech, rather than relying on a predetermined head pose from a template video.
We also introduce pose modelling in speech2latent for pose controllability.
Additionally, we propose a novel method for generating continuous video frames
with the DDIM image decoder trained on individual frames, eliminating the need
for modelling the joint distribution of consecutive frames directly. Our
experiments show that DAE-Talker outperforms existing popular methods in
lip-sync, video fidelity, and pose naturalness. We also conduct ablation
studies to analyze the effectiveness of the proposed techniques and demonstrate
the pose controllability of DAE-Talker.
