  We propose a robust and accurate method for estimating the 3D poses of two
hands in close interaction from a single color image. This is a very
challenging problem, as large occlusions and many confusions between the joints
may happen. State-of-the-art methods solve this problem by regressing a heatmap
for each joint, which requires solving two problems simultaneously: localizing
the joints and recognizing them. In this work, we propose to separate these
tasks by relying on a CNN to first localize joints as 2D keypoints, and on
self-attention between the CNN features at these keypoints to associate them
with the corresponding hand joint. The resulting architecture, which we call
"Keypoint Transformer", is highly efficient as it achieves state-of-the-art
performance with roughly half the number of model parameters on the
InterHand2.6M dataset. We also show it can be easily extended to estimate the
3D pose of an object manipulated by one or two hands with high performance.
Moreover, we created a new dataset of more than 75,000 images of two hands
manipulating an object fully annotated in 3D and will make it publicly
available.
