  Tracking the full skeletal pose of the hands and fingers is a challenging
problem that has a plethora of applications for user interaction. Existing
techniques either require wearable hardware, add restrictions to user pose, or
require significant computation resources. This research explores a new
approach to tracking hands, or any articulated model, by using an augmented
rigid body simulation. This allows us to phrase 3D object tracking as a linear
complementarity problem with a well-defined solution. Based on a depth sensor's
samples, the system generates constraints that limit motion orthogonal to the
rigid body model's surface. These constraints, along with prior motion,
collision/contact constraints, and joint mechanics, are resolved with a
projected Gauss-Seidel solver. Due to camera noise properties and attachment
errors, the numerous surface constraints are impulse capped to avoid
overpowering mechanical constraints. To improve tracking accuracy, multiple
simulations are spawned at each frame and fed a variety of heuristics,
constraints and poses. A 3D error metric selects the best-fit simulation,
helping the system handle challenging hand motions. Such an approach enables
real-time, robust, and accurate 3D skeletal tracking of a user's hand on a
variety of depth cameras, while only utilizing a single x86 CPU core for
processing.
