  3D hand shape and pose estimation from a single depth map is a new and
challenging computer vision problem with many applications. Existing methods
addressing it directly regress hand meshes via 2D convolutional neural
networks, which leads to artefacts due to perspective distortions in the
images. To address the limitations of the existing methods, we develop
HandVoxNet++, i.e., a voxel-based deep network with 3D and graph convolutions
trained in a fully supervised manner. The input to our network is a 3D
voxelized-depth-map-based on the truncated signed distance function (TSDF).
HandVoxNet++ relies on two hand shape representations. The first one is the 3D
voxelized grid of hand shape, which does not preserve the mesh topology and
which is the most accurate representation. The second representation is the
hand surface that preserves the mesh topology. We combine the advantages of
both representations by aligning the hand surface to the voxelized hand shape
either with a new neural Graph-Convolutions-based Mesh Registration
(GCN-MeshReg) or classical segment-wise Non-Rigid Gravitational Approach
(NRGA++) which does not rely on training data. In extensive evaluations on
three public benchmarks, i.e., SynHand5M, depth-based HANDS19 challenge and
HO-3D, the proposed HandVoxNet++ achieves state-of-the-art performance. In this
journal extension of our previous approach presented at CVPR 2020, we gain
41.09% and 13.7% higher shape alignment accuracy on SynHand5M and HANDS19
datasets, respectively. Our method is ranked first on the HANDS19 challenge
dataset (Task 1: Depth-Based 3D Hand Pose Estimation) at the moment of the
submission of our results to the portal in August 2020.
