  It is still an interesting and challenging problem to synthesize a vivid and
realistic singing face driven by music signal. In this paper, we present a
method for this task with natural motions of the lip, facial expression, head
pose, and eye states. Due to the coupling of the mixed information of human
voice and background music in common signals of music audio, we design a
decouple-and-fuse strategy to tackle the challenge. We first decompose the
input music audio into human voice stream and background music stream. Due to
the implicit and complicated correlation between the two-stream input signals
and the dynamics of the facial expressions, head motions and eye states, we
model their relationship with an attention scheme, where the effects of the two
streams are fused seamlessly. Furthermore, to improve the expressiveness of the
generated results, we propose to decompose head movements generation into speed
generation and direction generation, and decompose eye states generation into
the short-time eye blinking generation and the long-time eye closing generation
to model them separately. We also build a novel SingingFace Dataset to support
the training and evaluation of this task, and to facilitate future works on
this topic. Extensive experiments and user study show that our proposed method
is capable of synthesizing vivid singing face, which is better than
state-of-the-art methods qualitatively and quantitatively.
