  Large-scale driving datasets such as Waymo Open Dataset and nuScenes
substantially accelerate autonomous driving research, especially for perception
tasks such as 3D detection and trajectory forecasting. Since the driving logs
in these datasets contain HD maps and detailed object annotations which
accurately reflect the real-world complexity of traffic behaviors, we can
harvest a massive number of complex traffic scenarios and recreate their
digital twins in simulation. Compared to the hand-crafted scenarios often used
in existing simulators, data-driven scenarios collected from the real world can
facilitate many research opportunities in machine learning and autonomous
driving. In this work, we present ScenarioNet, an open-source platform for
large-scale traffic scenario modeling and simulation. ScenarioNet defines a
unified scenario description format and collects a large-scale repository of
real-world traffic scenarios from the heterogeneous data in various driving
datasets including Waymo, nuScenes, Lyft L5, and nuPlan datasets. These
scenarios can be further replayed and interacted with in multiple views from
Bird-Eye-View layout to realistic 3D rendering in MetaDrive simulator. This
provides a benchmark for evaluating the safety of autonomous driving stacks in
simulation before their real-world deployment. We further demonstrate the
strengths of ScenarioNet on large-scale scenario generation, imitation
learning, and reinforcement learning in both single-agent and multi-agent
settings. Code, demo videos, and website are available at
https://metadriverse.github.io/scenarionet.
