  A long-standing goal in scene understanding is to obtain interpretable and
editable representations that can be directly constructed from a raw monocular
RGB-D video, without requiring specialized hardware setup or priors. The
problem is significantly more challenging in the presence of multiple moving
and/or deforming objects. Traditional methods have approached the setup with a
mix of simplifications, scene priors, pretrained templates, or known
deformation models. The advent of neural representations, especially neural
implicit representations and radiance fields, opens the possibility of
end-to-end optimization to collectively capture geometry, appearance, and
object motion. However, current approaches produce global scene encoding,
assume multiview capture with limited or no motion in the scenes, and do not
facilitate easy manipulation beyond novel view synthesis. In this work, we
introduce a factored neural scene representation that can directly be learned
from a monocular RGB-D video to produce object-level neural presentations with
an explicit encoding of object movement (e.g., rigid trajectory) and/or
deformations (e.g., nonrigid movement). We evaluate ours against a set of
neural approaches on both synthetic and real data to demonstrate that the
representation is efficient, interpretable, and editable (e.g., change object
trajectory). Code and data are available at
http://geometry.cs.ucl.ac.uk/projects/2023/factorednerf .
