  Scalable sensor simulation is an important yet challenging open problem for
safety-critical domains such as self-driving. Current works in image simulation
either fail to be photorealistic or do not model the 3D environment and the
dynamic objects within, losing high-level control and physical realism. In this
paper, we present GeoSim, a geometry-aware image composition process which
synthesizes novel urban driving scenarios by augmenting existing images with
dynamic objects extracted from other scenes and rendered at novel poses.
Towards this goal, we first build a diverse bank of 3D objects with both
realistic geometry and appearance from sensor data. During simulation, we
perform a novel geometry-aware simulation-by-composition procedure which 1)
proposes plausible and realistic object placements into a given scene, 2)
render novel views of dynamic objects from the asset bank, and 3) composes and
blends the rendered image segments. The resulting synthetic images are
realistic, traffic-aware, and geometrically consistent, allowing our approach
to scale to complex use cases. We demonstrate two such important applications:
long-range realistic video simulation across multiple camera sensors, and
synthetic data generation for data augmentation on downstream segmentation
tasks. Please check https://tmux.top/publication/geosim/ for high-resolution
video results.
