  This paper introduces the first text-guided work for generating the sequence
of hand-object interaction in 3D. The main challenge arises from the lack of
labeled data where existing ground-truth datasets are nowhere near
generalizable in interaction type and object category, which inhibits the
modeling of diverse 3D hand-object interaction with the correct physical
implication (e.g., contacts and semantics) from text prompts. To address this
challenge, we propose to decompose the interaction generation task into two
subtasks: hand-object contact generation; and hand-object motion generation.
For contact generation, a VAE-based network takes as input a text and an object
mesh, and generates the probability of contacts between the surfaces of hands
and the object during the interaction. The network learns a variety of local
geometry structure of diverse objects that is independent of the objects'
category, and thus, it is applicable to general objects. For motion generation,
a Transformer-based diffusion model utilizes this 3D contact map as a strong
prior for generating physically plausible hand-object motion as a function of
text prompts by learning from the augmented labeled dataset; where we annotate
text labels from many existing 3D hand and object motion data. Finally, we
further introduce a hand refiner module that minimizes the distance between the
object surface and hand joints to improve the temporal stability of the
object-hand contacts and to suppress the penetration artifacts. In the
experiments, we demonstrate that our method can generate more realistic and
diverse interactions compared to other baseline methods. We also show that our
method is applicable to unseen objects. We will release our model and newly
labeled data as a strong foundation for future research. Codes and data are
available in: https://github.com/JunukCha/Text2HOI.
