  Relighting is an essential step in realistically transferring objects from a
captured image into another environment. For example, authentic telepresence in
Augmented Reality requires faces to be displayed and relit consistent with the
observer's scene lighting. We investigate end-to-end deep learning
architectures that both de-light and relight an image of a human face. Our
model decomposes the input image into intrinsic components according to a
diffuse physics-based image formation model. We enable non-diffuse effects
including cast shadows and specular highlights by predicting a residual
correction to the diffuse render. To train and evaluate our model, we collected
a portrait database of 21 subjects with various expressions and poses. Each
sample is captured in a controlled light stage setup with 32 individual light
sources. Our method creates precise and believable relighting results and
generalizes to complex illumination conditions and challenging poses, including
when the subject is not looking straight at the camera.
