  Recent success of vision foundation models have shown promising performance
for the 2D perception tasks. However, it is difficult to train a 3D foundation
network directly due to the limited dataset and it remains under explored
whether existing foundation models can be lifted to 3D space seamlessly. In
this paper, we present PointSeg, a novel training-free paradigm that leverages
off-the-shelf vision foundation models to address 3D scene perception tasks.
PointSeg can segment anything in 3D scene by acquiring accurate 3D prompts to
align their corresponding pixels across frames. Concretely, we design a
two-branch prompts learning structure to construct the 3D point-box prompts
pairs, combining with the bidirectional matching strategy for accurate point
and proposal prompts generation. Then, we perform the iterative post-refinement
adaptively when cooperated with different vision foundation models. Moreover,
we design a affinity-aware merging algorithm to improve the final ensemble
masks. PointSeg demonstrates impressive segmentation performance across various
datasets, all without training. Specifically, our approach significantly
surpasses the state-of-the-art specialist training-free model by 14.1$\%$,
12.3$\%$, and 12.6$\%$ mAP on ScanNet, ScanNet++, and KITTI-360 datasets,
respectively. On top of that, PointSeg can incorporate with various foundation
models and even surpasses the specialist training-based methods by
3.4$\%$-5.4$\%$ mAP across various datasets, serving as an effective generalist
model.
