  Thanks to its precise spatial referencing, 3D point cloud visual grounding is
essential for deep understanding and dynamic interaction in 3D environments,
encompassing 3D Referring Expression Comprehension (3DREC) and Segmentation
(3DRES). We argue that 3DREC and 3DRES should be unified in one framework,
which is also a natural progression in the community. To explain, 3DREC help
3DRES locate the referent, while 3DRES also facilitate 3DREC via more
fine-grained language-visual alignment. To achieve this, this paper takes the
initiative step to integrate 3DREC and 3DRES into a unified framework, termed
3D Referring Transformer (3DRefTR). Its key idea is to build upon a mature
3DREC model and leverage ready query embeddings and visual tokens from the
3DREC model to construct a dedicated mask branch. Specially, we propose
Superpoint Mask Branch, which serves a dual purpose: i) By harnessing on the
inherent association between the superpoints and point cloud, it eliminates the
heavy computational overhead on the high-resolution visual features for
upsampling; ii) By leveraging the heterogeneous CPU-GPU parallelism, while the
GPU is occupied generating visual and language tokens, the CPU concurrently
produces superpoints, equivalently accomplishing the upsampling computation.
This elaborate design enables 3DRefTR to achieve both well-performing 3DRES and
3DREC capacities with only a 6% additional latency compared to the original
3DREC model. Empirical evaluations affirm the superiority of 3DRefTR.
Specifically, on the ScanRefer dataset, 3DRefTR surpasses the state-of-the-art
3DRES method by 12.43% in mIoU and improves upon the SOTA 3DREC method by 0.6%
<EMAIL_ADDRESS>The codes and models will be released soon.
