  While previous speech-driven talking face generation methods have made
significant progress in improving the visual quality and lip-sync quality of
the synthesized videos, they pay less attention to lip motion jitters which
greatly undermine the realness of talking face videos. What causes motion
jitters, and how to mitigate the problem? In this paper, we conduct systematic
analyses on the motion jittering problem based on a state-of-the-art pipeline
that uses 3D face representations to bridge the input audio and output video,
and improve the motion stability with a series of effective designs. We find
that several issues can lead to jitters in synthesized talking face video: 1)
jitters from the input 3D face representations; 2) training-inference mismatch;
3) lack of dependency modeling among video frames. Accordingly, we propose
three effective solutions to address this issue: 1) we propose a gaussian-based
adaptive smoothing module to smooth the 3D face representations to eliminate
jitters in the input; 2) we add augmented erosions on the input data of the
neural renderer in training to simulate the distortion in inference to reduce
mismatch; 3) we develop an audio-fused transformer generator to model
dependency among video frames. Besides, considering there is no off-the-shelf
metric for measuring motion jitters in talking face video, we devise an
objective metric (Motion Stability Index, MSI), to quantitatively measure the
motion jitters by calculating the reciprocal of variance acceleration.
Extensive experimental results show the superiority of our method on
motion-stable face video generation, with better quality than previous systems.
