  A recent trend among generalizable novel view synthesis methods is to learn a
rendering operator acting over single camera rays. This approach is promising
because it removes the need for explicit volumetric rendering, but it
effectively treats target images as collections of independent pixels. Here, we
propose to learn a global rendering operator acting over all camera rays
jointly. We show that the right representation to enable such rendering is a
5-dimensional plane sweep volume consisting of the projection of the input
images on a set of planes facing the target camera. Based on this
understanding, we introduce our Convolutional Global Latent Renderer (ConvGLR),
an efficient convolutional architecture that performs the rendering operation
globally in a low-resolution latent space. Experiments on various datasets
under sparse and generalizable setups show that our approach consistently
outperforms existing methods by significant margins.
