  Music-to-dance translation is a brand-new and powerful feature in recent
role-playing games. Players can now let their characters dance along with
specified music clips and even generate fan-made dance videos. Previous works
of this topic consider music-to-dance as a supervised motion generation problem
based on time-series data. However, these methods suffer from limited training
data pairs and the degradation of movements. This paper provides a new
perspective for this task where we re-formulate the translation problem as a
piece-wise dance phrase retrieval problem based on the choreography theory.
With such a design, players are allowed to further edit the dance movements on
top of our generation while other regression based methods ignore such user
interactivity. Considering that the dance motion capture is an expensive and
time-consuming procedure which requires the assistance of professional dancers,
we train our method under a semi-supervised learning framework with a large
unlabeled dataset (20x than labeled data) collected. A co-ascent mechanism is
introduced to improve the robustness of our network. Using this unlabeled
dataset, we also introduce self-supervised pre-training so that the translator
can understand the melody, rhythm, and other components of music phrases. We
show that the pre-training significantly improves the translation accuracy than
that of training from scratch. Experimental results suggest that our method not
only generalizes well over various styles of music but also succeeds in
expert-level choreography for game players.
