  Neural rendering-based urban scene reconstruction methods commonly rely on
images collected from driving vehicles with cameras facing and moving forward.
Although these methods can successfully synthesize from views similar to
training camera trajectory, directing the novel view outside the training
camera distribution does not guarantee on-par performance. In this paper, we
tackle the Extrapolated View Synthesis (EVS) problem by evaluating the
reconstructions on views such as looking left, right or downwards with respect
to training camera distributions. To improve rendering quality for EVS, we
initialize our model by constructing dense LiDAR map, and propose to leverage
prior scene knowledge such as surface normal estimator and large-scale
diffusion model. Qualitative and quantitative comparisons demonstrate the
effectiveness of our methods on EVS. To the best of our knowledge, we are the
first to address the EVS problem in urban scene reconstruction. Link to our
project page: https://vegs3d.github.io/.
