  In this paper, we introduce a method to automatically reconstruct the 3D
motion of a person interacting with an object from a single RGB video. Our
method estimates the 3D poses of the person together with the object pose, the
contact positions and the contact forces exerted on the human body. The main
contributions of this work are three-fold. First, we introduce an approach to
jointly estimate the motion and the actuation forces of the person on the
manipulated object by modeling contacts and the dynamics of the interactions.
This is cast as a large-scale trajectory optimization problem. Second, we
develop a method to automatically recognize from the input video the 2D
position and timing of contacts between the person and the object or the
ground, thereby significantly simplifying the complexity of the optimization.
Third, we validate our approach on a recent video+MoCap dataset capturing
typical parkour actions, and demonstrate its performance on a new dataset of
Internet videos showing people manipulating a variety of tools in unconstrained
environments.
