  Although deep neural networks hold the state-of-the-art in several remote
sensing tasks, their black-box operation hinders the understanding of their
decisions, concealing any bias and other shortcomings in datasets and model
performance. To this end, we have applied explainable artificial intelligence
(XAI) methods in remote sensing multi-label classification tasks towards
producing human-interpretable explanations and improve transparency. In
particular, we utilized and trained deep learning models with state-of-the-art
performance in the benchmark BigEarthNet and SEN12MS datasets. Ten XAI methods
were employed towards understanding and interpreting models' predictions, along
with quantitative metrics to assess and compare their performance. Numerous
experiments were performed to assess the overall performance of XAI methods for
straightforward prediction cases, competing multiple labels, as well as
misclassification cases. According to our findings, Occlusion, Grad-CAM and
Lime were the most interpretable and reliable XAI methods. However, none
delivers high-resolution outputs, while apart from Grad-CAM, both Lime and
Occlusion are computationally expensive. We also highlight different aspects of
XAI performance and elaborate with insights on black-box decisions in order to
improve transparency, understand their behavior and reveal, as well, datasets'
particularities.
