  Reconstructing human-object interaction in 3D from a single RGB image is a
challenging task and existing data driven methods do not generalize beyond the
objects present in the carefully curated 3D interaction datasets. Capturing
large-scale real data to learn strong interaction and 3D shape priors is very
expensive due to the combinatorial nature of human-object interactions. In this
paper, we propose ProciGen (Procedural interaction Generation), a method to
procedurally generate datasets with both, plausible interaction and diverse
object variation. We generate 1M+ human-object interaction pairs in 3D and
leverage this large-scale data to train our HDM (Hierarchical Diffusion Model),
a novel method to reconstruct interacting human and unseen objects, without any
templates. Our HDM is an image-conditioned diffusion model that learns both
realistic interaction and highly accurate human and object shapes. Experiments
show that our HDM trained with ProciGen significantly outperforms prior methods
that requires template meshes and that our dataset allows training methods with
strong generalization ability to unseen object instances. Our code and data are
released.
