  We show how to relight a scene, depicted in a single image, such that (a) the
overall shading has changed and (b) the resulting image looks like a natural
image of that scene. Applications for such a procedure include generating
training data and building authoring environments. Naive methods for doing this
fail. One reason is that shading and albedo are quite strongly related; for
example, sharp boundaries in shading tend to appear at depth discontinuities,
which usually apparent in albedo. The same scene can be lit in different ways,
and established theory shows the different lightings form a cone (the
illumination cone). Novel theory shows that one can use similar scenes to
estimate the different lightings that apply to a given scene, with bounded
expected error. Our method exploits this theory to estimate a representation of
the available lighting fields in the form of imputed generators of the
illumination cone. Our procedure does not require expensive "inverse graphics"
datasets, and sees no ground truth data of any kind.
  Qualitative evaluation suggests the method can erase and restore soft indoor
shadows, and can "steer" light around a scene. We offer a summary quantitative
evaluation of the method with a novel application of the FID. An extension of
the FID allows per-generated-image evaluation. Furthermore, we offer
qualitative evaluation with a user study, and show that our method produces
images that can successfully be used for data augmentation.
