  Panoptic reconstruction is a challenging task in 3D scene understanding.
However, most existing methods heavily rely on pre-trained semantic
segmentation models and known 3D object bounding boxes for 3D panoptic
segmentation, which is not available for in-the-wild scenes. In this paper, we
propose a novel zero-shot panoptic reconstruction method from RGB-D images of
scenes. For zero-shot segmentation, we leverage open-vocabulary instance
segmentation, but it has to face partial labeling and instance association
challenges. We tackle both challenges by propagating partial labels with the
aid of dense generalized features and building a 3D instance graph for
associating 2D instance IDs. Specifically, we exploit partial labels to learn a
classifier for generalized semantic features to provide complete labels for
scenes with dense distilled features. Moreover, we formulate instance
association as a 3D instance graph segmentation problem, allowing us to fully
utilize the scene geometry prior and all 2D instance masks to infer global
unique pseudo 3D instance ID. Our method outperforms state-of-the-art methods
on the indoor dataset ScanNet V2 and the outdoor dataset KITTI-360,
demonstrating the effectiveness of our graph segmentation method and
reconstruction network.
