  Novel view synthesis is a long-standing problem. In this work, we consider a
variant of the problem where we are given only a few context views sparsely
covering a scene or an object. The goal is to predict novel viewpoints in the
scene, which requires learning priors. The current state of the art is based on
Neural Radiance Field (NeRF), and while achieving impressive results, the
methods suffer from long training times as they require evaluating millions of
3D point samples via a neural network for each image. We propose a 2D-only
method that maps multiple context views and a query pose to a new image in a
single pass of a neural network. Our model uses a two-stage architecture
consisting of a codebook and a transformer model. The codebook is used to embed
individual images into a smaller latent space, and the transformer solves the
view synthesis task in this more compact space. To train our model efficiently,
we introduce a novel branching attention mechanism that allows us to use the
same model not only for neural rendering but also for camera pose estimation.
Experimental results on real-world scenes show that our approach is competitive
compared to NeRF-based methods while not reasoning explicitly in 3D, and it is
faster to train.
