  We study the problem of imitating object interactions from Internet videos.
This requires understanding the hand-object interactions in 4D, spatially in 3D
and over time, which is challenging due to mutual hand-object occlusions. In
this paper we make two main contributions: (1) a novel reconstruction technique
RHOV (Reconstructing Hands and Objects from Videos), which reconstructs 4D
trajectories of both the hand and the object using 2D image cues and temporal
smoothness constraints; (2) a system for imitating object interactions in a
physics simulator with reinforcement learning. We apply our reconstruction
technique to 100 challenging Internet videos. We further show that we can
successfully imitate a range of different object interactions in a physics
simulator. Our object-centric approach is not limited to human-like
end-effectors and can learn to imitate object interactions using different
embodiments, like a robotic arm with a parallel jaw gripper.
