  Multimodal image fusion and object detection are crucial for autonomous
driving. While current methods have advanced the fusion of texture details and
semantic information, their complex training processes hinder broader
applications. Addressing this challenge, we introduce E2E-MFD, a novel
end-to-end algorithm for multimodal fusion detection. E2E-MFD streamlines the
process, achieving high performance with a single training phase. It employs
synchronous joint optimization across components to avoid suboptimal solutions
tied to individual tasks. Furthermore, it implements a comprehensive
optimization strategy in the gradient matrix for shared parameters, ensuring
convergence to an optimal fusion detection configuration. Our extensive testing
on multiple public datasets reveals E2E-MFD's superior capabilities, showcasing
not only visually appealing image fusion but also impressive detection
outcomes, such as a 3.9% and 2.0% mAP50 increase on horizontal object detection
dataset M3FD and oriented object detection dataset DroneVehicle, respectively,
compared to state-of-the-art approaches. The code is released at
https://github.com/icey-zhang/E2E-MFD.
