  Audio-driven talking face generation, which aims to synthesize talking faces
with realistic facial animations (including accurate lip movements, vivid
facial expression details and natural head poses) corresponding to the audio,
has achieved rapid progress in recent years. However, most existing work
focuses on generating lip movements only without handling the closely
correlated facial expressions, which degrades the realism of the generated
faces greatly. This paper presents DIRFA, a novel method that can generate
talking faces with diverse yet realistic facial animations from the same
driving audio. To accommodate fair variation of plausible facial animations for
the same audio, we design a transformer-based probabilistic mapping network
that can model the variational facial animation distribution conditioned upon
the input audio and autoregressively convert the audio signals into a facial
animation sequence. In addition, we introduce a temporally-biased mask into the
mapping network, which allows to model the temporal dependency of facial
animations and produce temporally smooth facial animation sequence. With the
generated facial animation sequence and a source image, photo-realistic talking
faces can be synthesized with a generic generation network. Extensive
experiments show that DIRFA can generate talking faces with realistic facial
animations effectively.
