  Decomposing a scene into its shape, reflectance, and illumination is a
challenging but important problem in computer vision and graphics. This problem
is inherently more challenging when the illumination is not a single light
source under laboratory conditions but is instead an unconstrained
environmental illumination. Though recent work has shown that implicit
representations can be used to model the radiance field of an object, most of
these techniques only enable view synthesis and not relighting. Additionally,
evaluating these radiance fields is resource and time-intensive. We propose a
neural reflectance decomposition (NeRD) technique that uses physically-based
rendering to decompose the scene into spatially varying BRDF material
properties. In contrast to existing techniques, our input images can be
captured under different illumination conditions. In addition, we also propose
techniques to convert the learned reflectance volume into a relightable
textured mesh enabling fast real-time rendering with novel illuminations. We
demonstrate the potential of the proposed approach with experiments on both
synthetic and real datasets, where we are able to obtain high-quality
relightable 3D assets from image collections. The datasets and code is
available on the project page: https://markboss.me/publication/2021-nerd/
