  Simulation systems have become an essential component in the development and
validation of autonomous driving technologies. The prevailing state-of-the-art
approach for simulation is to use game engines or high-fidelity computer
graphics (CG) models to create driving scenarios. However, creating CG models
and vehicle movements (e.g., the assets for simulation) remains a manual task
that can be costly and time-consuming. In addition, the fidelity of CG images
still lacks the richness and authenticity of real-world images and using these
images for training leads to degraded performance.
  In this paper we present a novel approach to address these issues: Augmented
Autonomous Driving Simulation (AADS). Our formulation augments real-world
pictures with a simulated traffic flow to create photo-realistic simulation
images and renderings. More specifically, we use LiDAR and cameras to scan
street scenes. From the acquired trajectory data, we generate highly plausible
traffic flows for cars and pedestrians and compose them into the background.
The composite images can be re-synthesized with different viewpoints and sensor
models. The resulting images are photo-realistic, fully annotated, and ready
for end-to-end training and testing of autonomous driving systems from
perception to planning. We explain our system design and validate our
algorithms with a number of autonomous driving tasks from detection to
segmentation and predictions.
  Compared to traditional approaches, our method offers unmatched scalability
and realism. Scalability is particularly important for AD simulation and we
believe the complexity and diversity of the real world cannot be realistically
captured in a virtual environment. Our augmented approach combines the
flexibility in a virtual environment (e.g., vehicle movements) with the
richness of the real world to allow effective simulation of anywhere on earth.
