  As RGB-D sensors become more affordable, using RGB-D images to obtain
high-accuracy 6D pose estimation results becomes a better option.
State-of-the-art approaches typically use different backbones to extract
features for RGB and depth images. They use a 2D CNN for RGB images and a
per-pixel point cloud network for depth data, as well as a fusion network for
feature fusion. We find that the essential reason for using two independent
backbones is the "projection breakdown" problem. In the depth image plane, the
projected 3D structure of the physical world is preserved by the 1D depth value
and its built-in 2D pixel coordinate (UV). Any spatial transformation that
modifies UV, such as resize, flip, crop, or pooling operations in the CNN
pipeline, breaks the binding between the pixel value and UV coordinate. As a
consequence, the 3D structure is no longer preserved by a modified depth image
or feature. To address this issue, we propose a simple yet effective method
denoted as Uni6D that explicitly takes the extra UV data along with RGB-D
images as input. Our method has a Unified CNN framework for 6D pose estimation
with a single CNN backbone. In particular, the architecture of our method is
based on Mask R-CNN with two extra heads, one named RT head for directly
predicting 6D pose and the other named abc head for guiding the network to map
the visible points to their coordinates in the 3D model as an auxiliary module.
This end-to-end approach balances simplicity and accuracy, achieving comparable
accuracy with state of the arts and 7.2x faster inference speed on the
YCB-Video dataset.
