  We address the task of 6D multi-object pose: given a set of known 3D objects
and an RGB or RGB-D input image, we detect and estimate the 6D pose of each
object. We propose a new approach to 6D object pose estimation which consists
of an end-to-end differentiable architecture that makes use of geometric
knowledge. Our approach iteratively refines both pose and correspondence in a
tightly coupled manner, allowing us to dynamically remove outliers to improve
accuracy. We use a novel differentiable layer to perform pose refinement by
solving an optimization problem we refer to as Bidirectional Depth-Augmented
Perspective-N-Point (BD-PnP). Our method achieves state-of-the-art accuracy on
standard 6D Object Pose benchmarks. Code is available at
https://github.com/princeton-vl/Coupled-Iterative-Refinement.
