  Differentiable volumetric rendering is a powerful paradigm for 3D
reconstruction and novel view synthesis. However, standard volume rendering
approaches struggle with degenerate geometries in the case of limited viewpoint
diversity, a common scenario in robotics applications. In this work, we propose
to use the multi-view photometric objective from the self-supervised depth
estimation literature as a geometric regularizer for volumetric rendering,
significantly improving novel view synthesis without requiring additional
information. Building upon this insight, we explore the explicit modeling of
scene geometry using a generalist Transformer, jointly learning a radiance
field as well as depth and light fields with a set of shared latent codes. We
demonstrate that sharing geometric information across tasks is mutually
beneficial, leading to improvements over single-task learning without an
increase in network complexity. Our DeLiRa architecture achieves
state-of-the-art results on the ScanNet benchmark, enabling high quality
volumetric rendering as well as real-time novel view and depth synthesis in the
limited viewpoint diversity setting.
