  LiDAR and cameras are two complementary sensors for 3D perception in
autonomous driving. LiDAR point clouds have accurate spatial and geometry
information, while RGB images provide textural and color data for context
reasoning. To exploit LiDAR and cameras jointly, existing fusion methods tend
to align each 3D point to only one projected image pixel based on calibration,
namely one-to-one mapping. However, the performance of these approaches highly
relies on the calibration quality, which is sensitive to the temporal and
spatial synchronization of sensors. Therefore, we propose a Dynamic Cross
Attention (DCA) module with a novel one-to-many cross-modality mapping that
learns multiple offsets from the initial projection towards the neighborhood
and thus develops tolerance to calibration error. Moreover, a \textit{dynamic
query enhancement} is proposed to perceive the model-independent calibration,
which further strengthens DCA's tolerance to the initial misalignment. The
whole fusion architecture named Dynamic Cross Attention Network (DCAN) exploits
multi-level image features and adapts to multiple representations of point
clouds, which allows DCA to serve as a plug-in fusion module. Extensive
experiments on nuScenes and KITTI prove DCA's effectiveness. The proposed DCAN
outperforms state-of-the-art methods on the nuScenes detection challenge.
