  We propose a new dataset and a novel approach to learning hand-object
interaction priors for hand and articulated object pose estimation. We first
collect a dataset using visual teleoperation, where the human operator can
directly play within a physical simulator to manipulate the articulated
objects. We record the data and obtain free and accurate annotations on object
poses and contact information from the simulator. Our system only requires an
iPhone to record human hand motion, which can be easily scaled up and largely
lower the costs of data and annotation collection. With this data, we learn 3D
interaction priors including a discriminator (in a GAN) capturing the
distribution of how object parts are arranged, and a diffusion model which
generates the contact regions on articulated objects, guiding the hand pose
estimation. Such structural and contact priors can easily transfer to
real-world data with barely any domain gap. By using our data and learned
priors, our method significantly improves the performance on joint hand and
articulated object poses estimation over the existing state-of-the-art methods.
The project is available at https://zehaozhu.github.io/ContactArt/ .
