  Multimodal-driven talking face generation refers to animating a portrait with
the given pose, expression, and gaze transferred from the driving image and
video, or estimated from the text and audio. However, existing methods ignore
the potential of text modal, and their generators mainly follow the
source-oriented feature rearrange paradigm coupled with unstable GAN
frameworks. In this work, we first represent the emotion in the text prompt,
which could inherit rich semantics from the CLIP, allowing flexible and
generalized emotion control. We further reorganize these tasks as the
target-oriented texture transfer and adopt the Diffusion Models. More
specifically, given a textured face as the source and the rendered face
projected from the desired 3DMM coefficients as the target, our proposed
Texture-Geometry-aware Diffusion Model decomposes the complex transfer problem
into multi-conditional denoising process, where a Texture Attention-based
module accurately models the correspondences between appearance and geometry
cues contained in source and target conditions, and incorporate extra implicit
information for high-fidelity talking face generation. Additionally, TGDM can
be gracefully tailored for face swapping. We derive a novel paradigm free of
unstable seesaw-style optimization, resulting in simple, stable, and effective
training and inference schemes. Extensive experiments demonstrate the
superiority of our method.
