  Recent vision-language pre-training models have exhibited remarkable
generalization ability in zero-shot recognition tasks. Previous open-vocabulary
3D scene understanding methods mostly focus on training 3D models using either
image or text supervision while neglecting the collective strength of all
modalities. In this work, we propose a Dense Multimodal Alignment (DMA)
framework to densely co-embed different modalities into a common space for
maximizing their synergistic benefits. Instead of extracting coarse view- or
region-level text prompts, we leverage large vision-language models to extract
complete category information and scalable scene descriptions to build the text
modality, and take image modality as the bridge to build dense point-pixel-text
associations. Besides, in order to enhance the generalization ability of the 2D
model for downstream 3D tasks without compromising the open-vocabulary
capability, we employ a dual-path integration approach to combine frozen CLIP
visual features and learnable mask features. Extensive experiments show that
our DMA method produces highly competitive open-vocabulary segmentation
performance on various indoor and outdoor tasks.
