  While recent developments in autonomous vehicle (AV) technology highlight
substantial progress, we lack tools for rigorous and scalable testing.
Real-world testing, the $\textit{de facto}$ evaluation environment, places the
public in danger, and, due to the rare nature of accidents, will require
billions of miles in order to statistically validate performance claims. We
implement a simulation framework that can test an entire modern autonomous
driving system, including, in particular, systems that employ deep-learning
perception and control algorithms. Using adaptive importance-sampling methods
to accelerate rare-event probability evaluation, we estimate the probability of
an accident under a base distribution governing standard traffic behavior. We
demonstrate our framework on a highway scenario, accelerating system evaluation
by $2$-$20$ times over naive Monte Carlo sampling methods and $10$-$300
\mathsf{P}$ times (where $\mathsf{P}$ is the number of processors) over
real-world testing.
