  Synthetic data is a scalable alternative to manual supervision, but it
requires overcoming the sim-to-real domain gap. This discrepancy between
virtual and real worlds is addressed by two seemingly opposed approaches:
improving the realism of simulation or foregoing realism entirely via domain
randomization. In this paper, we show that the recent progress in neural
rendering enables a new unified approach we call Photo-realistic Neural Domain
Randomization (PNDR). We propose to learn a composition of neural networks that
acts as a physics-based ray tracer generating high-quality renderings from
scene geometry alone. Our approach is modular, composed of different neural
networks for materials, lighting, and rendering, thus enabling randomization of
different key image generation components in a differentiable pipeline. Once
trained, our method can be combined with other methods and used to generate
photo-realistic image augmentations online and significantly more efficiently
than via traditional ray-tracing. We demonstrate the usefulness of PNDR through
two downstream tasks: 6D object detection and monocular depth estimation. Our
experiments show that training with PNDR enables generalization to novel scenes
and significantly outperforms the state of the art in terms of real-world
transfer.
