  With the extremely rapid advances in remote sensing (RS) technology, a great
quantity of Earth observation (EO) data featuring considerable and complicated
heterogeneity is readily available nowadays, which renders researchers an
opportunity to tackle current geoscience applications in a fresh way. With the
joint utilization of EO data, much research on multimodal RS data fusion has
made tremendous progress in recent years, yet these developed traditional
algorithms inevitably meet the performance bottleneck due to the lack of the
ability to comprehensively analyse and interpret these strongly heterogeneous
data. Hence, this non-negligible limitation further arouses an intense demand
for an alternative tool with powerful processing competence. Deep learning
(DL), as a cutting-edge technology, has witnessed remarkable breakthroughs in
numerous computer vision tasks owing to its impressive ability in data
representation and reconstruction. Naturally, it has been successfully applied
to the field of multimodal RS data fusion, yielding great improvement compared
with traditional methods. This survey aims to present a systematic overview in
DL-based multimodal RS data fusion. More specifically, some essential knowledge
about this topic is first given. Subsequently, a literature survey is conducted
to analyse the trends of this field. Some prevalent sub-fields in the
multimodal RS data fusion are then reviewed in terms of the to-be-fused data
modalities, i.e., spatiospectral, spatiotemporal, light detection and
ranging-optical, synthetic aperture radar-optical, and RS-Geospatial Big Data
fusion. Furthermore, We collect and summarize some valuable resources for the
sake of the development in multimodal RS data fusion. Finally, the remaining
challenges and potential future directions are highlighted.
