  Current multi-modal object detection approaches focus on the vehicle domain
and are limited in the perception range and the processing capabilities.
Roadside sensor units (RSUs) introduce a new domain for perception systems and
leverage altitude to observe traffic. Cameras and LiDARs mounted on gantry
bridges increase the perception range and produce a full digital twin of the
traffic. In this work, we introduce InfraDet3D, a multi-modal 3D object
detector for roadside infrastructure sensors. We fuse two LiDARs using early
fusion and further incorporate detections from monocular cameras to increase
the robustness and to detect small objects. Our monocular 3D detection module
uses HD maps to ground object yaw hypotheses, improving the final perception
results. The perception framework is deployed on a real-world intersection that
is part of the A9 Test Stretch in Munich, Germany. We perform several ablation
studies and experiments and show that fusing two LiDARs with two cameras leads
to an improvement of +1.90 mAP compared to a camera-only solution. We evaluate
our results on the A9 infrastructure dataset and achieve 68.48 mAP on the test
set. The dataset and code will be available at https://a9-dataset.com to allow
the research community to further improve the perception results and make
autonomous driving safer.
