  Multi-sensor modal fusion has demonstrated strong advantages in 3D object
detection tasks. However, existing methods that fuse multi-modal features
require transforming features into the bird's eye view space and may lose
certain information on Z-axis, thus leading to inferior performance. To this
end, we propose a novel end-to-end multi-modal fusion transformer-based
framework, dubbed FusionFormer, that incorporates deformable attention and
residual structures within the fusion encoding module. Specifically, by
developing a uniform sampling strategy, our method can easily sample from 2D
image and 3D voxel features spontaneously, thus exploiting flexible
adaptability and avoiding explicit transformation to the bird's eye view space
during the feature concatenation process. We further implement a residual
structure in our feature encoder to ensure the model's robustness in case of
missing an input modality. Through extensive experiments on a popular
autonomous driving benchmark dataset, nuScenes, our method achieves
state-of-the-art single model performance of 72.6% mAP and 75.1% NDS in the 3D
object detection task without test time augmentation.
