  In this work, we present a novel data-driven method for robust 6DoF object
pose estimation from a single RGBD image. Unlike previous methods that directly
regressing pose parameters, we tackle this challenging task with a
keypoint-based approach. Specifically, we propose a deep Hough voting network
to detect 3D keypoints of objects and then estimate the 6D pose parameters
within a least-squares fitting manner. Our method is a natural extension of
2D-keypoint approaches that successfully work on RGB based 6DoF estimation. It
allows us to fully utilize the geometric constraint of rigid objects with the
extra depth information and is easy for a network to learn and optimize.
Extensive experiments were conducted to demonstrate the effectiveness of
3D-keypoint detection in the 6D pose estimation task. Experimental results also
show our method outperforms the state-of-the-art methods by large margins on
several benchmarks. Code and video are available at
https://github.com/ethnhe/PVN3D.git.
