  We aim to generate fine-grained 3D geometry from large-scale sparse LiDAR
scans, abundantly captured by autonomous vehicles (AV). Contrary to prior work
on AV scene completion, we aim to extrapolate fine geometry from unlabeled and
beyond spatial limits of LiDAR scans, taking a step towards generating
realistic, high-resolution simulation-ready 3D street environments. We propose
hierarchical Generative Cellular Automata (hGCA), a spatially scalable
conditional 3D generative model, which grows geometry recursively with local
kernels following, in a coarse-to-fine manner, equipped with a light-weight
planner to induce global consistency. Experiments on synthetic scenes show that
hGCA generates plausible scene geometry with higher fidelity and completeness
compared to state-of-the-art baselines. Our model generalizes strongly from
sim-to-real, qualitatively outperforming baselines on the Waymo-open dataset.
We also show anecdotal evidence of the ability to create novel objects from
real-world geometric cues even when trained on limited synthetic content. More
results and details can be found on
https://research.nvidia.com/labs/toronto-ai/hGCA/.
