  Monocular 3D motion capture (mocap) is beneficial to many applications. The
use of a single camera, however, often fails to handle occlusions of different
body parts and hence it is limited to capture relatively simple movements. We
present a light-weight, hybrid mocap technique called HybridCap that augments
the camera with only 4 Inertial Measurement Units (IMUs) in a
learning-and-optimization framework. We first employ a weakly-supervised and
hierarchical motion inference module based on cooperative Gated Recurrent Unit
(GRU) blocks that serve as limb, body and root trackers as well as an inverse
kinematics solver. Our network effectively narrows the search space of
plausible motions via coarse-to-fine pose estimation and manages to tackle
challenging movements with high efficiency. We further develop a hybrid
optimization scheme that combines inertial feedback and visual cues to improve
tracking accuracy. Extensive experiments on various datasets demonstrate
HybridCap can robustly handle challenging movements ranging from fitness
actions to Latin dance. It also achieves real-time performance up to 60 fps
with state-of-the-art accuracy.
