  Realistic reconstruction of two hands interacting with objects is a new and
challenging problem that is essential for building personalized Virtual and
Augmented Reality environments. Graph Convolutional networks (GCNs) allow for
the preservation of the topologies of hands poses and shapes by modeling them
as a graph. In this work, we propose the THOR-Net which combines the power of
GCNs, Transformer, and self-supervision to realistically reconstruct two hands
and an object from a single RGB image. Our network comprises two stages; namely
the features extraction stage and the reconstruction stage. In the features
extraction stage, a Keypoint RCNN is used to extract 2D poses, features maps,
heatmaps, and bounding boxes from a monocular RGB image. Thereafter, this 2D
information is modeled as two graphs and passed to the two branches of the
reconstruction stage. The shape reconstruction branch estimates meshes of two
hands and an object using our novel coarse-to-fine GraFormer shape network. The
3D poses of the hands and objects are reconstructed by the other branch using a
GraFormer network. Finally, a self-supervised photometric loss is used to
directly regress the realistic textured of each vertex in the hands' meshes.
Our approach achieves State-of-the-art results in Hand shape estimation on the
HO-3D dataset (10.0mm) exceeding ArtiBoost (10.8mm). It also surpasses other
methods in hand pose estimation on the challenging two hands and object (H2O)
dataset by 5mm on the left-hand pose and 1 mm on the right-hand pose.
