  Hand manipulating objects is an important interaction motion in our daily
activities. We faithfully reconstruct this motion with a single RGBD camera by
a novel deep reinforcement learning method to leverage physics. Firstly, we
propose object compensation control which establishes direct object control to
make the network training more stable. Meanwhile, by leveraging the
compensation force and torque, we seamlessly upgrade the simple point contact
model to a more physical-plausible surface contact model, further improving the
reconstruction accuracy and physical correctness. Experiments indicate that
without involving any heuristic physical rules, this work still successfully
involves physics in the reconstruction of hand-object interactions which are
complex motions hard to imitate with deep reinforcement learning. Our code and
data are available at https://github.com/hu-hy17/HOIC.
