  Neural Radiance Fields (NeRFs) have emerged as promising tools for advancing
autonomous driving (AD) research, offering scalable closed-loop simulation and
data augmentation capabilities. However, to trust the results achieved in
simulation, one needs to ensure that AD systems perceive real and rendered data
in the same way. Although the performance of rendering methods is increasing,
many scenarios will remain inherently challenging to reconstruct faithfully. To
this end, we propose a novel perspective for addressing the real-to-simulated
data gap. Rather than solely focusing on improving rendering fidelity, we
explore simple yet effective methods to enhance perception model robustness to
NeRF artifacts without compromising performance on real data. Moreover, we
conduct the first large-scale investigation into the real-to-simulated data gap
in an AD setting using a state-of-the-art neural rendering technique.
Specifically, we evaluate object detectors and an online mapping model on real
and simulated data, and study the effects of different fine-tuning
strategies.Our results show notable improvements in model robustness to
simulated data, even improving real-world performance in some cases. Last, we
delve into the correlation between the real-to-simulated gap and image
reconstruction metrics, identifying FID and LPIPS as strong indicators. See
https://research.zenseact.com/publications/closing-real2sim-gap for our project
page.
