  Photorealistic editing of outdoor scenes from photographs requires a profound
understanding of the image formation process and an accurate estimation of the
scene geometry, reflectance and illumination. A delicate manipulation of the
lighting can then be performed while keeping the scene albedo and geometry
unaltered. We present NeRF-OSR, i.e., the first approach for outdoor scene
relighting based on neural radiance fields. In contrast to the prior art, our
technique allows simultaneous editing of both scene illumination and camera
viewpoint using only a collection of outdoor photos shot in uncontrolled
settings. Moreover, it enables direct control over the scene illumination, as
defined through a spherical harmonics model. For evaluation, we collect a new
benchmark dataset of several outdoor sites photographed from multiple
viewpoints and at different times. For each time, a 360 degree environment map
is captured together with a colour-calibration chequerboard to allow accurate
numerical evaluations on real data against ground truth. Comparisons against
SoTA show that NeRF-OSR enables controllable lighting and viewpoint editing at
higher quality and with realistic self-shadowing reproduction. Our method and
the dataset are publicly available at https://4dqv.mpi-inf.mpg.de/NeRF-OSR/.
