  Recent advancements in perception for autonomous driving are driven by deep
learning. In order to achieve robust and accurate scene understanding,
autonomous vehicles are usually equipped with different sensors (e.g. cameras,
LiDARs, Radars), and multiple sensing modalities can be fused to exploit their
complementary properties. In this context, many methods have been proposed for
deep multi-modal perception problems. However, there is no general guideline
for network architecture design, and questions of "what to fuse", "when to
fuse", and "how to fuse" remain open. This review paper attempts to
systematically summarize methodologies and discuss challenges for deep
multi-modal object detection and semantic segmentation in autonomous driving.
To this end, we first provide an overview of on-board sensors on test vehicles,
open datasets, and background information for object detection and semantic
segmentation in autonomous driving research. We then summarize the fusion
methodologies and discuss challenges and open questions. In the appendix, we
provide tables that summarize topics and methods. We also provide an
interactive online platform to navigate each reference:
https://boschresearch.github.io/multimodalperception/.
