  We present a novel learning approach to recover the 6D poses and sizes of
unseen object instances from an RGB-D image. To handle the intra-class shape
variation, we propose a deep network to reconstruct the 3D object model by
explicitly modeling the deformation from a pre-learned categorical shape prior.
Additionally, our network infers the dense correspondences between the depth
observation of the object instance and the reconstructed 3D model to jointly
estimate the 6D object pose and size. We design an autoencoder that trains on a
collection of object models and compute the mean latent embedding for each
category to learn the categorical shape priors. Extensive experiments on both
synthetic and real-world datasets demonstrate that our approach significantly
outperforms the state of the art. Our code is available at
https://github.com/mentian/object-deformnet.
