  By identifying four important components of existing LiDAR-camera 3D object
detection methods (LiDAR and camera candidates, transformation, and fusion
outputs), we observe that all existing methods either find dense candidates or
yield dense representations of scenes. However, given that objects occupy only
a small part of a scene, finding dense candidates and generating dense
representations is noisy and inefficient. We propose SparseFusion, a novel
multi-sensor 3D detection method that exclusively uses sparse candidates and
sparse representations. Specifically, SparseFusion utilizes the outputs of
parallel detectors in the LiDAR and camera modalities as sparse candidates for
fusion. We transform the camera candidates into the LiDAR coordinate space by
disentangling the object representations. Then, we can fuse the multi-modality
candidates in a unified 3D space by a lightweight self-attention module. To
mitigate negative transfer between modalities, we propose novel semantic and
geometric cross-modality transfer modules that are applied prior to the
modality-specific detectors. SparseFusion achieves state-of-the-art performance
on the nuScenes benchmark while also running at the fastest speed, even
outperforming methods with stronger backbones. We perform extensive experiments
to demonstrate the effectiveness and efficiency of our modules and overall
method pipeline. Our code will be made publicly available at
https://github.com/yichen928/SparseFusion.
