  4D reconstruction of human-object interaction is critical for immersive VR/AR
experience and human activity understanding. Recent advances still fail to
recover fine geometry and texture results from sparse RGB inputs, especially
under challenging human-object interactions scenarios. In this paper, we
propose a neural human performance capture and rendering system to generate
both high-quality geometry and photo-realistic texture of both human and
objects under challenging interaction scenarios in arbitrary novel views, from
only sparse RGB streams. To deal with complex occlusions raised by human-object
interactions, we adopt a layer-wise scene decoupling strategy and perform
volumetric reconstruction and neural rendering of the human and object.
Specifically, for geometry reconstruction, we propose an interaction-aware
human-object capture scheme that jointly considers the human reconstruction and
object reconstruction with their correlations. Occlusion-aware human
reconstruction and robust human-aware object tracking are proposed for
consistent 4D human-object dynamic reconstruction. For neural texture
rendering, we propose a layer-wise human-object rendering scheme, which
combines direction-aware neural blending weight learning and spatial-temporal
texture completion to provide high-resolution and photo-realistic texture
results in the occluded scenarios. Extensive experiments demonstrate the
effectiveness of our approach to achieve high-quality geometry and texture
reconstruction in free viewpoints for challenging human-object interactions.
