A Scenario-Based Platform for Testing Autonomous Vehicle Behavior
  Prediction Models in Simulation
  Behavior prediction remains one of the most challenging tasks in the
autonomous vehicle (AV) software stack. Forecasting the future trajectories of
nearby agents plays a critical role in ensuring road safety, as it equips AVs
with the necessary information to plan safe routes of travel. However, these
prediction models are data-driven and trained on data collected in real life
that may not represent the full range of scenarios an AV can encounter. Hence,
it is important that these prediction models are extensively tested in various
test scenarios involving interactive behaviors prior to deployment. To support
this need, we present a simulation-based testing platform which supports (1)
intuitive scenario modeling with a probabilistic programming language called
Scenic, (2) specifying a multi-objective evaluation metric with a partial
priority ordering, (3) falsification of the provided metric, and (4)
parallelization of simulations for scalable testing. As a part of the platform,
we provide a library of 25 Scenic programs that model challenging test
scenarios involving interactive traffic participant behaviors. We demonstrate
the effectiveness and the scalability of our platform by testing a trained
behavior prediction model and searching for failure scenarios.
