  Recently, groundbreaking results have been presented on open-vocabulary
semantic image segmentation. Such methods segment each pixel in an image into
arbitrary categories provided at run-time in the form of text prompts, as
opposed to a fixed set of classes defined at training time. In this work, we
present a zero-shot volumetric open-vocabulary semantic scene segmentation
method. Our method builds on the insight that we can fuse image features from a
vision-language model into a neural implicit representation. We show that the
resulting feature field can be segmented into different classes by assigning
points to natural language text prompts. The implicit volumetric representation
enables us to segment the scene both in 3D and 2D by rendering feature maps
from any given viewpoint of the scene. We show that our method works on noisy
real-world data and can run in real-time on live sensor data dynamically
adjusting to text prompts. We also present quantitative comparisons on the
ScanNet dataset.
