  Simulation has the potential to transform the development of robust
algorithms for mobile agents deployed in safety-critical scenarios. However,
the poor photorealism and lack of diverse sensor modalities of existing
simulation engines remain key hurdles towards realizing this potential. Here,
we present VISTA, an open source, data-driven simulator that integrates
multiple types of sensors for autonomous vehicles. Using high fidelity,
real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and
event-based cameras, enabling the rapid generation of novel viewpoints in
simulation and thereby enriching the data available for policy learning with
corner cases that are difficult to capture in the physical world. Using VISTA,
we demonstrate the ability to train and test perception-to-control policies
across each of the sensor types and showcase the power of this approach via
deployment on a full scale autonomous vehicle. The policies learned in VISTA
exhibit sim-to-real transfer without modification and greater robustness than
those trained exclusively on real-world data.
