  This paper presents a neural rendering method for controllable portrait video
synthesis. Recent advances in volumetric neural rendering, such as neural
radiance fields (NeRF), has enabled the photorealistic novel view synthesis of
static scenes with impressive results. However, modeling dynamic and
controllable objects as part of a scene with such scene representations is
still challenging. In this work, we design a system that enables both novel
view synthesis for portrait video, including the human subject and the scene
background, and explicit control of the facial expressions through a
low-dimensional expression representation. We leverage the expression space of
a 3D morphable face model (3DMM) to represent the distribution of human facial
expressions, and use it to condition the NeRF volumetric function. Furthermore,
we impose a spatial prior brought by 3DMM fitting to guide the network to learn
disentangled control for scene appearance and facial actions. We demonstrate
the effectiveness of our method on free view synthesis of portrait videos with
expression controls. To train a scene, our method only requires a short video
of a subject captured by a mobile device.
