  Synthesizing natural interactions between virtual humans and their 3D
environments is critical for numerous applications, such as computer games and
AR/VR experiences. Our goal is to synthesize humans interacting with a given 3D
scene controlled by high-level semantic specifications as pairs of action
categories and object instances, e.g., "sit on the chair". The key challenge of
incorporating interaction semantics into the generation framework is to learn a
joint representation that effectively captures heterogeneous information,
including human body articulation, 3D object geometry, and the intent of the
interaction. To address this challenge, we design a novel transformer-based
generative model, in which the articulated 3D human body surface points and 3D
objects are jointly encoded in a unified latent space, and the semantics of the
interaction between the human and objects are embedded via positional encoding.
Furthermore, inspired by the compositional nature of interactions that humans
can simultaneously interact with multiple objects, we define interaction
semantics as the composition of varying numbers of atomic action-object pairs.
Our proposed generative model can naturally incorporate varying numbers of
atomic interactions, which enables synthesizing compositional human-scene
interactions without requiring composite interaction data. We extend the PROX
dataset with interaction semantic labels and scene instance segmentation to
evaluate our method and demonstrate that our method can generate realistic
human-scene interactions with semantic control. Our perceptual study shows that
our synthesized virtual humans can naturally interact with 3D scenes,
considerably outperforming existing methods. We name our method COINS, for
COmpositional INteraction Synthesis with Semantic Control. Code and data are
available at https://github.com/zkf1997/COINS.
