  3D referring segmentation is an emerging and challenging vision-language task
that aims to segment the object described by a natural language expression in a
point cloud scene. The key challenge behind this task is vision-language
feature fusion and alignment. In this work, we propose RefMask3D to explore the
comprehensive multi-modal feature interaction and understanding. First, we
propose a Geometry-Enhanced Group-Word Attention to integrate language with
geometrically coherent sub-clouds through cross-modal group-word attention,
which effectively addresses the challenges posed by the sparse and irregular
nature of point clouds. Then, we introduce a Linguistic Primitives Construction
to produce semantic primitives representing distinct semantic attributes, which
greatly enhance the vision-language understanding at the decoding stage.
Furthermore, we introduce an Object Cluster Module that analyzes the
interrelationships among linguistic primitives to consolidate their insights
and pinpoint common characteristics, helping to capture holistic information
and enhance the precision of target identification. The proposed RefMask3D
achieves new state-of-the-art performance on 3D referring segmentation, 3D
visual grounding, and also 2D referring image segmentation. Especially,
RefMask3D outperforms previous state-of-the-art method by a large margin of
3.16% mIoU} on the challenging ScanRefer dataset. Code is available at
https://github.com/heshuting555/RefMask3D.
