  This work addresses the problem of generating 3D holistic body motions from
human speech. Given a speech recording, we synthesize sequences of 3D body
poses, hand gestures, and facial expressions that are realistic and diverse. To
achieve this, we first build a high-quality dataset of 3D holistic body meshes
with synchronous speech. We then define a novel speech-to-motion generation
framework in which the face, body, and hands are modeled separately. The
separated modeling stems from the fact that face articulation strongly
correlates with human speech, while body poses and hand gestures are less
correlated. Specifically, we employ an autoencoder for face motions, and a
compositional vector-quantized variational autoencoder (VQ-VAE) for the body
and hand motions. The compositional VQ-VAE is key to generating diverse
results. Additionally, we propose a cross-conditional autoregressive model that
generates body poses and hand gestures, leading to coherent and realistic
motions. Extensive experiments and user studies demonstrate that our proposed
approach achieves state-of-the-art performance both qualitatively and
quantitatively. Our novel dataset and code will be released for research
purposes at https://talkshow.is.tue.mpg.de.
