  The automated synthesis of high-quality 3D gestures from speech is of
significant value in virtual humans and gaming. Previous methods focus on
synthesizing gestures that are synchronized with speech rhythm, yet they
frequently overlook the inclusion of semantic gestures. These are sparse and
follow a long-tailed distribution across the gesture sequence, making them
difficult to learn in an end-to-end manner. Moreover, generating gestures,
rhythmically aligned with speech, faces a significant issue that cannot be
generalized to in-the-wild speeches. To address these issues, we introduce
SIGGesture, a novel diffusion-based approach for synthesizing realistic
gestures that are of both high quality and semantically pertinent.
Specifically, we firstly build a strong diffusion-based foundation model for
rhythmical gesture synthesis by pre-training it on a collected large-scale
dataset with pseudo labels. Secondly, we leverage the powerful generalization
capabilities of Large Language Models (LLMs) to generate proper semantic
gestures for the various speech content. Finally, we propose a semantic
injection module to infuse semantic information into the synthesized results
during diffusion reverse process. Extensive experiments demonstrate that the
proposed SIGGesture significantly outperforms existing baselines and shows
excellent generalization and controllability.
