  The task of 3D semantic scene graph (3DSSG) prediction in the point cloud is
challenging since (1) the 3D point cloud only captures geometric structures
with limited semantics compared to 2D images, and (2) long-tailed relation
distribution inherently hinders the learning of unbiased prediction. Since 2D
images provide rich semantics and scene graphs are in nature coped with
languages, in this study, we propose Visual-Linguistic Semantics Assisted
Training (VL-SAT) scheme that can significantly empower 3DSSG prediction models
with discrimination about long-tailed and ambiguous semantic relations. The key
idea is to train a powerful multi-modal oracle model to assist the 3D model.
This oracle learns reliable structural representations based on semantics from
vision, language, and 3D geometry, and its benefits can be heterogeneously
passed to the 3D model during the training stage. By effectively utilizing
visual-linguistic semantics in training, our VL-SAT can significantly boost
common 3DSSG prediction models, such as SGFN and SGGpoint, only with 3D inputs
in the inference stage, especially when dealing with tail relation triplets.
Comprehensive evaluations and ablation studies on the 3DSSG dataset have
validated the effectiveness of the proposed scheme. Code is available at
https://github.com/wz7in/CVPR2023-VLSAT.
