  Although the remote sensing (RS) community has begun to pretrain transformers
(intended to be fine-tuned on RS tasks), it is unclear how these models perform
under distribution shifts. Here, we pretrain a new RS transformer--called
SatViT-V2--on 1.3 million satellite-derived RS images, then fine-tune it (along
with five other models) to investigate how it performs on distributions not
seen during training. We split an expertly labeled land cover dataset into 14
datasets based on source biome. We train each model on each biome separately
and test them on all other biomes. In all, this amounts to 1638 biome transfer
experiments. After fine-tuning, we find that SatViT-V2 outperforms SatViT-V1 by
3.1% on in-distribution (matching biomes) and 2.8% on out-of-distribution
(mismatching biomes) data. Additionally, we find that initializing fine-tuning
from the linear probed solution (i.e., leveraging LPFT [1]) improves
SatViT-V2's performance by another 1.2% on in-distribution and 2.4% on
out-of-distribution data. Next, we find that pretrained RS transformers are
better calibrated under distribution shifts than non-pretrained models and
leveraging LPFT results in further improvements in model calibration. Lastly,
we find that five measures of distribution shift are moderately correlated with
biome transfer performance. We share code and pretrained model weights.
(https://github.com/antofuller/SatViT)
