  Recently, digital humans for interpersonal interaction in virtual
environments have gained significant attention. In this paper, we introduce a
novel multi-dancer synthesis task called partner dancer generation, which
involves synthesizing virtual human dancers capable of performing dance with
users. The task aims to control the pose diversity between the lead dancer and
the partner dancer. The core of this task is to ensure the controllable
diversity of the generated partner dancer while maintaining temporal
coordination with the lead dancer. This scenario varies from earlier research
in generating dance motions driven by music, as our emphasis is on
automatically designing partner dancer postures according to pre-defined
diversity, the pose of lead dancer, as well as the accompanying tunes. To
achieve this objective, we propose a three-stage framework called
Dance-with-You (DanY). Initially, we employ a 3D Pose Collection stage to
collect a wide range of basic dance poses as references for motion generation.
Then, we introduce a hyper-parameter that coordinates the similarity between
dancers by masking poses to prevent the generation of sequences that are
over-diverse or consistent. To avoid the rigidity of movements, we design a
Dance Pre-generated stage to pre-generate these masked poses instead of filling
them with zeros. After that, a Dance Motion Transfer stage is adopted with
leader sequences and music, in which a multi-conditional sampling formula is
rewritten to transfer the pre-generated poses into a sequence with a partner
style. In practice, to address the lack of multi-person datasets, we introduce
AIST-M, a new dataset for partner dancer generation, which is publicly
availiable. Comprehensive evaluations on our AIST-M dataset demonstrate that
the proposed DanY can synthesize satisfactory partner dancer results with
controllable diversity.
