  This paper presents the multi-modal BigEarthNet (BigEarthNet-MM) benchmark
archive made up of 590,326 pairs of Sentinel-1 and Sentinel-2 image patches to
support the deep learning (DL) studies in multi-modal multi-label remote
sensing (RS) image retrieval and classification. Each pair of patches in
BigEarthNet-MM is annotated with multi-labels provided by the CORINE Land Cover
(CLC) map of 2018 based on its thematically most detailed Level-3 class
nomenclature. Our initial research demonstrates that some CLC classes are
challenging to be accurately described by only considering (single-date)
BigEarthNet-MM images. In this paper, we also introduce an alternative
class-nomenclature as an evolution of the original CLC labels to address this
problem. This is achieved by interpreting and arranging the CLC Level-3
nomenclature based on the properties of BigEarthNet-MM images in a new
nomenclature of 19 classes. In our experiments, we show the potential of
BigEarthNet-MM for multi-modal multi-label image retrieval and classification
problems by considering several state-of-the-art DL models. We also demonstrate
that the DL models trained from scratch on BigEarthNet-MM outperform those
pre-trained on ImageNet, especially in relation to some complex classes,
including agriculture and other vegetated and natural environments. We make all
the data and the DL models publicly available at https://bigearth.net, offering
an important resource to support studies on multi-modal image scene
classification and retrieval problems in RS.
