  We tackle the problem of producing realistic simulations of LiDAR point
clouds, the sensor of preference for most self-driving vehicles. We argue that,
by leveraging real data, we can simulate the complex world more realistically
compared to employing virtual worlds built from CAD/procedural models. Towards
this goal, we first build a large catalog of 3D static maps and 3D dynamic
objects by driving around several cities with our self-driving fleet. We can
then generate scenarios by selecting a scene from our catalog and "virtually"
placing the self-driving vehicle (SDV) and a set of dynamic objects from the
catalog in plausible locations in the scene. To produce realistic simulations,
we develop a novel simulator that captures both the power of physics-based and
learning-based simulation. We first utilize ray casting over the 3D scene and
then use a deep neural network to produce deviations from the physics-based
simulation, producing realistic LiDAR point clouds. We showcase LiDARsim's
usefulness for perception algorithms-testing on long-tail events and end-to-end
closed-loop evaluation on safety-critical scenarios.
