  In this paper, we present a novel, scalable approach for constructing open
set, instance-level 3D scene representations, advancing open world
understanding of 3D environments. Existing methods require pre-constructed 3D
scenes and face scalability issues due to per-point feature vector learning,
limiting their efficacy with complex queries. Our method overcomes these
limitations by incrementally building instance-level 3D scene representations
using 2D foundation models, efficiently aggregating instance-level details such
as masks, feature vectors, names, and captions. We introduce fusion schemes for
feature vectors to enhance their contextual knowledge and performance on
complex queries. Additionally, we explore large language models for robust
automatic annotation and spatial reasoning tasks. We evaluate our proposed
approach on multiple scenes from ScanNet and Replica datasets demonstrating
zero-shot generalization capabilities, exceeding current state-of-the-art
methods in open world 3D scene understanding.
