  We investigate the utility of in-domain self-supervised pre-training of
vision models in the analysis of remote sensing imagery. Self-supervised
learning (SSL) has emerged as a promising approach for remote sensing image
classification due to its ability to exploit large amounts of unlabeled data.
Unlike traditional supervised learning, SSL aims to learn representations of
data without the need for explicit labels. This is achieved by formulating
auxiliary tasks that can be used for pre-training models before fine-tuning
them on a given downstream task. A common approach in practice to SSL
pre-training is utilizing standard pre-training datasets, such as ImageNet.
While relevant, such a general approach can have a sub-optimal influence on the
downstream performance of models, especially on tasks from challenging domains
such as remote sensing. In this paper, we analyze the effectiveness of SSL
pre-training by employing the iBOT framework coupled with Vision transformers
trained on Million-AID, a large and unlabeled remote sensing dataset. We
present a comprehensive study of different self-supervised pre-training
strategies and evaluate their effect across 14 downstream datasets with diverse
properties. Our results demonstrate that leveraging large in-domain datasets
for self-supervised pre-training consistently leads to improved predictive
downstream performance, compared to the standard approaches found in practice.
