  Recent deep learning-based methods outperform traditional learning methods on
remote sensing (RS) semantic segmentation/classification tasks. However, they
require large training datasets and are generally known for lack of
transferability due to the highly disparate RS image content across different
geographical regions. Yet, there is no comprehensive analysis of their
transferability, i.e., to which extent a model trained on a source domain can
be readily applicable to a target domain. Therefore, in this paper, we aim to
investigate the raw transferability of traditional and deep learning (DL)
models, as well as the effectiveness of domain adaptation (DA) approaches in
enhancing the transferability of the DL models (adapted transferability). By
utilizing four highly diverse RS datasets, we train six models with and without
three DA approaches to analyze their transferability between these datasets
quantitatively. Furthermore, we developed a straightforward method to quantify
the transferability of a model using the spectral indices as a medium and have
demonstrated its effectiveness in evaluating the model transferability at the
target domain when the labels are unavailable. Our experiments yield several
generally important yet not well-reported observations regarding the raw and
adapted transferability. Moreover, our proposed label-free transferability
assessment method is validated to be better than posterior model confidence.
The findings can guide the future development of generalized RS learning
models. The trained models are released under this link:
https://github.com/GDAOSU/Transferability-Remote-Sensing
