  Deep learning has largely reshaped remote sensing (RS) research for aerial
image understanding and made a great success. Nevertheless, most of the
existing deep models are initialized with the ImageNet pretrained weights.
Since natural images inevitably present a large domain gap relative to aerial
images, probably limiting the finetuning performance on downstream aerial scene
tasks. This issue motivates us to conduct an empirical study of remote sensing
pretraining (RSP) on aerial images. To this end, we train different networks
from scratch with the help of the largest RS scene recognition dataset up to
now -- MillionAID, to obtain a series of RS pretrained backbones, including
both convolutional neural networks (CNN) and vision transformers such as Swin
and ViTAE, which have shown promising performance on computer vision tasks.
Then, we investigate the impact of RSP on representative downstream tasks
including scene recognition, semantic segmentation, object detection, and
change detection using these CNN and vision transformer backbones. Empirical
study shows that RSP can help deliver distinctive performances in scene
recognition tasks and in perceiving RS related semantics such as "Bridge" and
"Airplane". We also find that, although RSP mitigates the data discrepancies of
traditional ImageNet pretraining on RS images, it may still suffer from task
discrepancies, where downstream tasks require different representations from
scene recognition tasks. These findings call for further research efforts on
both large-scale pretraining datasets and effective pretraining methods. The
codes and pretrained models will be released at
https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing.
