  Comprehensive modeling of the surrounding 3D world is key to the success of
autonomous driving. However, existing perception tasks like object detection,
road structure segmentation, depth & elevation estimation, and open-set object
localization each only focus on a small facet of the holistic 3D scene
understanding task. This divide-and-conquer strategy simplifies the algorithm
development procedure at the cost of losing an end-to-end unified solution to
the problem. In this work, we address this limitation by studying camera-based
3D panoptic segmentation, aiming to achieve a unified occupancy representation
for camera-only 3D scene understanding. To achieve this, we introduce a novel
method called PanoOcc, which utilizes voxel queries to aggregate spatiotemporal
information from multi-frame and multi-view images in a coarse-to-fine scheme,
integrating feature learning and scene representation into a unified occupancy
representation. We have conducted extensive ablation studies to verify the
effectiveness and efficiency of the proposed method. Our approach achieves new
state-of-the-art results for camera-based semantic segmentation and panoptic
segmentation on the nuScenes dataset. Furthermore, our method can be easily
extended to dense occupancy prediction and has shown promising performance on
the Occ3D benchmark. The code will be released at
https://github.com/Robertwyq/PanoOcc.
