  We extend neural radiance fields (NeRFs) to dynamic large-scale urban scenes.
Prior work tends to reconstruct single video clips of short durations (up to 10
seconds). Two reasons are that such methods (a) tend to scale linearly with the
number of moving objects and input videos because a separate model is built for
each and (b) tend to require supervision via 3D bounding boxes and panoptic
labels, obtained manually or via category-specific models. As a step towards
truly open-world reconstructions of dynamic cities, we introduce two key
innovations: (a) we factorize the scene into three separate hash table data
structures to efficiently encode static, dynamic, and far-field radiance
fields, and (b) we make use of unlabeled target signals consisting of RGB
images, sparse LiDAR, off-the-shelf self-supervised 2D descriptors, and most
importantly, 2D optical flow.
  Operationalizing such inputs via photometric, geometric, and feature-metric
reconstruction losses enables SUDS to decompose dynamic scenes into the static
background, individual objects, and their motions. When combined with our
multi-branch table representation, such reconstructions can be scaled to tens
of thousands of objects across 1.2 million frames from 1700 videos spanning
geospatial footprints of hundreds of kilometers, (to our knowledge) the largest
dynamic NeRF built to date.
  We present qualitative initial results on a variety of tasks enabled by our
representations, including novel-view synthesis of dynamic urban scenes,
unsupervised 3D instance segmentation, and unsupervised 3D cuboid detection. To
compare to prior work, we also evaluate on KITTI and Virtual KITTI 2,
surpassing state-of-the-art methods that rely on ground truth 3D bounding box
annotations while being 10x quicker to train.
