  Remote sensing images present unique challenges to image analysis due to the
extensive geographic coverage, hardware limitations, and misaligned multi-scale
images. This paper revisits the classical multi-scale representation learning
problem but under the general framework of self-supervised learning for remote
sensing image understanding. We present Cross-Scale MAE, a self-supervised
model built upon the Masked Auto-Encoder (MAE).During pre-training, Cross-Scale
MAE employs scale augmentation techniques and enforces cross-scale consistency
constraints through both contrastive and generative losses to ensure consistent
and meaningful representations well-suited for a wide range of downstream
tasks. Further, our implementation leverages the xFormers library to accelerate
network pre-training on a single GPU while maintaining the quality of learned
representations. Experimental evaluations demonstrate that Cross-Scale MAE
exhibits superior performance compared to standard MAE and other
state-of-the-art remote sensing MAE methods.
