  Most recent multispectral object detectors employ a two-branch structure to
extract features from RGB and thermal images. While the two-branch structure
achieves better performance than a single-branch structure, it overlooks
inference efficiency. This conflict is increasingly aggressive, as recent works
solely pursue higher performance rather than both performance and efficiency.
In this paper, we address this issue by improving the performance of efficient
single-branch structures. We revisit the reasons causing the performance gap
between these structures. For the first time, we reveal the information
interference problem in the naive early-fusion strategy adopted by previous
single-branch structures. Besides, we find that the domain gap between
multispectral images, and weak feature representation of the single-branch
structure are also key obstacles for performance. Focusing on these three
problems, we propose corresponding solutions, including a novel shape-priority
early-fusion strategy, a weakly supervised learning method, and a core
knowledge distillation technique. Experiments demonstrate that single-branch
networks equipped with these three contributions achieve significant
performance enhancements while retaining high efficiency. Our code will be
available at
\url{https://github.com/XueZ-phd/Efficient-RGB-T-Early-Fusion-Detection}.
