  This work focuses on the 3D reconstruction of non-rigid objects based on
monocular RGB video sequences. Concretely, we aim at building high-fidelity
models for generic object categories and casually captured scenes. To this end,
we do not assume known root poses of objects, and do not utilize
category-specific templates or dense pose priors. The key idea of our method,
Root Pose Decomposition (RPD), is to maintain a per-frame root pose
transformation, meanwhile building a dense field with local transformations to
rectify the root pose. The optimization of local transformations is performed
by point registration to the canonical space. We also adapt RPD to multi-object
scenarios with object occlusions and individual differences. As a result, RPD
allows non-rigid 3D reconstruction for complicated scenarios containing objects
with large deformations, complex motion patterns, occlusions, and scale
diversities of different individuals. Such a pipeline potentially scales to
diverse sets of objects in the wild. We experimentally show that RPD surpasses
state-of-the-art methods on the challenging DAVIS, OVIS, and AMA datasets.
