  We present a multimodal learning-based method to simultaneously synthesize
co-speech facial expressions and upper-body gestures for digital characters
using RGB video data captured using commodity cameras. Our approach learns from
sparse face landmarks and upper-body joints, estimated directly from video
data, to generate plausible emotive character motions. Given a speech audio
waveform and a token sequence of the speaker's face landmark motion and
body-joint motion computed from a video, our method synthesizes the motion
sequences for the speaker's face landmarks and body joints to match the content
and the affect of the speech. We design a generator consisting of a set of
encoders to transform all the inputs into a multimodal embedding space
capturing their correlations, followed by a pair of decoders to synthesize the
desired face and pose motions. To enhance the plausibility of synthesis, we use
an adversarial discriminator that learns to differentiate between the face and
pose motions computed from the original videos and our synthesized motions
based on their affective expressions. To evaluate our approach, we extend the
TED Gesture Dataset to include view-normalized, co-speech face landmarks in
addition to body gestures. We demonstrate the performance of our method through
thorough quantitative and qualitative experiments on multiple evaluation
metrics and via a user study. We observe that our method results in low
reconstruction error and produces synthesized samples with diverse facial
expressions and body gestures for digital characters.
