  We present a neural-based model for relighting a half-body portrait image by
simply referring to another portrait image with the desired lighting condition.
Rather than following classical inverse rendering methodology that involves
estimating normals, albedo and environment maps, we implicitly encode the
subject and lighting in a latent space, and use these latent codes to generate
relighted images by neural rendering. A key technical innovation is the use of
a novel overcomplete lighting representation, which facilitates lighting
interpolation in the latent space, as well as helping regularize the
self-organization of the lighting latent space during training. In addition, we
propose a novel multiplicative neural render that more effectively combines the
subject and lighting latent codes for rendering. We also created a large-scale
photorealistic rendered relighting dataset for training, which allows our model
to generalize well to real images. Extensive experiments demonstrate that our
system not only outperforms existing methods for referral-based portrait
relighting, but also has the capability generate sequences of relighted images
via lighting rotations.
