  We propose StyleTalker, a novel audio-driven talking head generation model
that can synthesize a video of a talking person from a single reference image
with accurately audio-synced lip shapes, realistic head poses, and eye blinks.
Specifically, by leveraging a pretrained image generator and an image encoder,
we estimate the latent codes of the talking head video that faithfully reflects
the given audio. This is made possible with several newly devised components:
1) A contrastive lip-sync discriminator for accurate lip synchronization, 2) A
conditional sequential variational autoencoder that learns the latent motion
space disentangled from the lip movements, such that we can independently
manipulate the motions and lip movements while preserving the identity. 3) An
auto-regressive prior augmented with normalizing flow to learn a complex
audio-to-motion multi-modal latent space. Equipped with these components,
StyleTalker can generate talking head videos not only in a motion-controllable
way when another motion source video is given but also in a completely
audio-driven manner by inferring realistic motions from the input audio.
Through extensive experiments and user studies, we show that our model is able
to synthesize talking head videos with impressive perceptual quality which are
accurately lip-synced with the input audios, largely outperforming
state-of-the-art baselines.
