  We present a differentiable rendering framework for material and lighting
estimation from multi-view images and a reconstructed geometry. In the
framework, we represent scene lightings as the Neural Incident Light Field
(NeILF) and material properties as the surface BRDF modelled by multi-layer
perceptrons. Compared with recent approaches that approximate scene lightings
as the 2D environment map, NeILF is a fully 5D light field that is capable of
modelling illuminations of any static scenes. In addition, occlusions and
indirect lights can be handled naturally by the NeILF representation without
requiring multiple bounces of ray tracing, making it possible to estimate
material properties even for scenes with complex lightings and geometries. We
also propose a smoothness regularization and a Lambertian assumption to reduce
the material-lighting ambiguity during the optimization. Our method strictly
follows the physically-based rendering equation, and jointly optimizes material
and lighting through the differentiable rendering process. We have intensively
evaluated the proposed method on our in-house synthetic dataset, the DTU MVS
dataset, and real-world BlendedMVS scenes. Our method is able to outperform
previous methods by a significant margin in terms of novel view rendering
quality, setting a new state-of-the-art for image-based material and lighting
estimation.
