  3D object detection is a critical task in autonomous driving. Recently
multi-modal fusion-based 3D object detection methods, which combine the
complementary advantages of LiDAR and camera, have shown great performance
improvements over mono-modal methods. However, so far, no methods have
attempted to utilize the instance-level contextual image semantics to guide the
3D object detection. In this paper, we propose a simple and effective Painting
Adaptive Instance-prior for 3D object detection (PAI3D) to fuse instance-level
image semantics flexibly with point cloud features. PAI3D is a multi-modal
sequential instance-level fusion framework. It first extracts instance-level
semantic information from images, the extracted information, including objects
categorical label, point-to-object membership and object position, are then
used to augment each LiDAR point in the subsequent 3D detection network to
guide and improve detection performance. PAI3D outperforms the state-of-the-art
with a large margin on the nuScenes dataset, achieving 71.4 in mAP and 74.2 in
NDS on the test split. Our comprehensive experiments show that instance-level
image semantics contribute the most to the performance gain, and PAI3D works
well with any good-quality instance segmentation models and any modern point
cloud 3D encoders, making it a strong candidate for deployment on autonomous
vehicles.
