  In this paper, we investigate the use of diffusion models which are
pre-trained on large-scale image-caption pairs for open-vocabulary 3D semantic
understanding. We propose a novel method, namely Diff2Scene, which leverages
frozen representations from text-image generative models, along with
salient-aware and geometric-aware masks, for open-vocabulary 3D semantic
segmentation and visual grounding tasks. Diff2Scene gets rid of any labeled 3D
data and effectively identifies objects, appearances, materials, locations and
their compositions in 3D scenes. We show that it outperforms competitive
baselines and achieves significant improvements over state-of-the-art methods.
In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by
12%.
