  Many popular tourist landmarks are captured in a multitude of online, public
photos. These photos represent a sparse and unstructured sampling of the
plenoptic function for a particular scene. In this paper,we present a new
approach to novel view synthesis under time-varying illumination from such
data. Our approach builds on the recent multi-plane image (MPI) format for
representing local light fields under fixed viewing conditions. We introduce a
new DeepMPI representation, motivated by observations on the sparsity structure
of the plenoptic function, that allows for real-time synthesis of
photorealistic views that are continuous in both space and across changes in
lighting. Our method can synthesize the same compelling parallax and
view-dependent effects as previous MPI methods, while simultaneously
interpolating along changes in reflectance and illumination with time. We show
how to learn a model of these effects in an unsupervised way from an
unstructured collection of photos without temporal registration, demonstrating
significant improvements over recent work in neural rendering. More information
can be found crowdsampling.io.
