  LiDAR and camera, as two different sensors, supply geometric (point clouds)
and semantic (RGB images) information of 3D scenes. However, it is still
challenging for existing methods to fuse data from the two cross sensors,
making them complementary for quality 3D object detection (3OD). We propose
ImLiDAR, a new 3OD paradigm to narrow the cross-sensor discrepancies by
progressively fusing the multi-scale features of camera Images and LiDAR point
clouds. ImLiDAR enables to provide the detection head with cross-sensor yet
robustly fused features. To achieve this, two core designs exist in ImLiDAR.
First, we propose a cross-sensor dynamic message propagation module to combine
the best of the multi-scale image and point features. Second, we raise a direct
set prediction problem that allows designing an effective set-based detector to
tackle the inconsistency of the classification and localization confidences,
and the sensitivity of hand-tuned hyperparameters. Besides, the novel set-based
detector can be detachable and easily integrated into various detection
networks. Comparisons on both the KITTI and SUN-RGBD datasets show clear visual
and numerical improvements of our ImLiDAR over twenty-three state-of-the-art
3OD methods.
