  Thoroughly testing autonomy systems is crucial in the pursuit of safe
autonomous driving vehicles. It necessitates creating safety-critical scenarios
that go beyond what can be safely collected from real-world data, as many of
these scenarios occur infrequently on public roads. However, the evaluation of
most existing NVS methods relies on sporadic sampling of image frames from the
training data, comparing the rendered images with ground truth images using
metrics. Unfortunately, this evaluation protocol falls short of meeting the
actual requirements in closed-loop simulations. Specifically, the true
application demands the capability to render novel views that extend beyond the
original trajectory (such as cross-lane views), which are challenging to
capture in the real world. To address this, this paper presents a novel driving
view synthesis dataset and benchmark specifically designed for autonomous
driving simulations. This dataset is unique as it includes testing images
captured by deviating from the training trajectory by 1-4 meters. It comprises
six sequences encompassing various time and weather conditions. Each sequence
contains 450 training images, 150 testing images, and their corresponding
camera poses and intrinsic parameters. Leveraging this novel dataset, we
establish the first realistic benchmark for evaluating existing NVS approaches
under front-only and multi-camera settings. The experimental findings
underscore the significant gap that exists in current approaches, revealing
their inadequate ability to fulfill the demanding prerequisites of cross-lane
or closed-loop simulation. Our dataset is released publicly at the project
page: https://3d-aigc.github.io/XLD/.
