  Audio-driven talking head generation has drawn growing attention. To produce
talking head videos with desired facial expressions, previous methods rely on
extra reference videos to provide expression information, which may be
difficult to find and hence limits their usage. In this work, we propose
TalkCLIP, a framework that can generate talking heads where the expressions are
specified by natural language, hence allowing for specifying expressions more
conveniently. To model the mapping from text to expressions, we first construct
a text-video paired talking head dataset where each video has diverse text
descriptions that depict both coarse-grained emotions and fine-grained facial
movements. Leveraging the proposed dataset, we introduce a CLIP-based style
encoder that projects natural language-based descriptions to the
representations of expressions. TalkCLIP can even infer expressions for
descriptions unseen during training. TalkCLIP can also use text to modulate
expression intensity and edit expressions. Extensive experiments demonstrate
that TalkCLIP achieves the advanced capability of generating photo-realistic
talking heads with vivid facial expressions guided by text descriptions.
