  Accurate detection of obstacles in 3D is an essential task for autonomous
driving and intelligent transportation. In this work, we propose a general
multimodal fusion framework FusionPainting to fuse the 2D RGB image and 3D
point clouds at a semantic level for boosting the 3D object detection task.
Especially, the FusionPainting framework consists of three main modules: a
multi-modal semantic segmentation module, an adaptive attention-based semantic
fusion module, and a 3D object detector. First, semantic information is
obtained for 2D images and 3D Lidar point clouds based on 2D and 3D
segmentation approaches. Then the segmentation results from different sensors
are adaptively fused based on the proposed attention-based semantic fusion
module. Finally, the point clouds painted with the fused semantic label are
sent to the 3D detector for obtaining the 3D objection results. The
effectiveness of the proposed framework has been verified on the large-scale
nuScenes detection benchmark by comparing it with three different baselines.
The experimental results show that the fusion strategy can significantly
improve the detection performance compared to the methods using only point
clouds, and the methods using point clouds only painted with 2D segmentation
information. Furthermore, the proposed approach outperforms other
state-of-the-art methods on the nuScenes testing benchmark.
