  Real-time simultaneous tracking of hands manipulating and interacting with
external objects has many potential applications in augmented reality, tangible
computing, and wearable computing. However, due to difficult occlusions, fast
motions, and uniform hand appearance, jointly tracking hand and object pose is
more challenging than tracking either of the two separately. Many previous
approaches resort to complex multi-camera setups to remedy the occlusion
problem and often employ expensive segmentation and optimization steps which
makes real-time tracking impossible. In this paper, we propose a real-time
solution that uses a single commodity RGB-D camera. The core of our approach is
a 3D articulated Gaussian mixture alignment strategy tailored to hand-object
tracking that allows fast pose optimization. The alignment energy uses novel
regularizers to address occlusions and hand-object contacts. For added
robustness, we guide the optimization with discriminative part classification
of the hand and segmentation of the object. We conducted extensive experiments
on several existing datasets and introduce a new annotated hand-object dataset.
Quantitative and qualitative results show the key advantages of our method:
speed, accuracy, and robustness.
