  When people deliver a speech, they naturally move heads, and this rhythmic
head motion conveys prosodic information. However, generating a lip-synced
video while moving head naturally is challenging. While remarkably successful,
existing works either generate still talkingface videos or rely on
landmark/video frames as sparse/dense mapping guidance to generate head
movements, which leads to unrealistic or uncontrollable video synthesis. To
overcome the limitations, we propose a 3D-aware generative network along with a
hybrid embedding module and a non-linear composition module. Through modeling
the head motion and facial expressions1 explicitly, manipulating 3D animation
carefully, and embedding reference images dynamically, our approach achieves
controllable, photo-realistic, and temporally coherent talking-head videos with
natural head movements. Thoughtful experiments on several standard benchmarks
demonstrate that our method achieves significantly better results than the
state-of-the-art methods in both quantitative and qualitative comparisons. The
code is available on https://github.com/
lelechen63/Talking-head-Generation-with-Rhythmic-Head-Motion.
