  This paper proposes a new method for live free-viewpoint human performance
capture with dynamic details (e.g., cloth wrinkles) using a single RGBD camera.
Our main contributions are: (i) a multi-layer representation of garments and
body, and (ii) a physics-based performance capture procedure. We first digitize
the performer using multi-layer surface representation, which includes the
undressed body surface and separate clothing meshes. For performance capture,
we perform skeleton tracking, cloth simulation, and iterative depth fitting
sequentially for the incoming frame. By incorporating cloth simulation into the
performance capture pipeline, we can simulate plausible cloth dynamics and
cloth-body interactions even in the occluded regions, which was not possible in
previous capture methods. Moreover, by formulating depth fitting as a physical
process, our system produces cloth tracking results consistent with the depth
observation while still maintaining physical constraints. Results and
evaluations show the effectiveness of our method. Our method also enables new
types of applications such as cloth retargeting, free-viewpoint video rendering
and animations.
