  Inferring representations of 3D scenes from 2D observations is a fundamental
problem of computer graphics, computer vision, and artificial intelligence.
Emerging 3D-structured neural scene representations are a promising approach to
3D scene understanding. In this work, we propose a novel neural scene
representation, Light Field Networks or LFNs, which represent both geometry and
appearance of the underlying 3D scene in a 360-degree, four-dimensional light
field parameterized via a neural implicit representation. Rendering a ray from
an LFN requires only a single network evaluation, as opposed to hundreds of
evaluations per ray for ray-marching or volumetric based renderers in
3D-structured neural scene representations. In the setting of simple scenes, we
leverage meta-learning to learn a prior over LFNs that enables multi-view
consistent light field reconstruction from as little as a single image
observation. This results in dramatic reductions in time and memory complexity,
and enables real-time rendering. The cost of storing a 360-degree light field
via an LFN is two orders of magnitude lower than conventional methods such as
the Lumigraph. Utilizing the analytical differentiability of neural implicit
representations and a novel parameterization of light space, we further
demonstrate the extraction of sparse depth maps from LFNs.
