  Fusing LiDAR and camera information is essential for achieving accurate and
reliable 3D object detection in autonomous driving systems. This is challenging
due to the difficulty of combining multi-granularity geometric and semantic
features from two drastically different modalities. Recent approaches aim at
exploring the semantic densities of camera features through lifting points in
2D camera images (referred to as seeds) into 3D space, and then incorporate 2D
semantics via cross-modal interaction or fusion techniques. However, depth
information is under-investigated in these approaches when lifting points into
3D space, thus 2D semantics can not be reliably fused with 3D points. Moreover,
their multi-modal fusion strategy, which is implemented as concatenation or
attention, either can not effectively fuse 2D and 3D information or is unable
to perform fine-grained interactions in the voxel space. To this end, we
propose a novel framework with better utilization of the depth information and
fine-grained cross-modal interaction between LiDAR and camera, which consists
of two important components. First, a Multi-Depth Unprojection (MDU) method
with depth-aware designs is used to enhance the depth quality of the lifted
points at each interaction level. Second, a Gated Modality-Aware Convolution
(GMA-Conv) block is applied to modulate voxels involved with the camera
modality in a fine-grained manner and then aggregate multi-modal features into
a unified space. Together they provide the detection head with more
comprehensive features from LiDAR and camera. On the nuScenes test benchmark,
our proposed method, abbreviated as MSMDFusion, achieves state-of-the-art 3D
object detection results with 71.5% mAP and 74.0% NDS, and strong tracking
results with 74.0% AMOTA without using test-time-augmentation and ensemble
techniques. The code is available at https://github.com/SxJyJay/MSMDFusion.
