  Advances in high resolution remote sensing image analysis are currently
hampered by the difficulty of gathering enough annotated data for training deep
learning methods, giving rise to a variety of small datasets and associated
dataset-specific methods. Moreover, typical tasks such as classification and
retrieval lack a systematic evaluation on standard benchmarks and training
datasets, which make it hard to identify durable and generalizable scientific
contributions. We aim at unifying remote sensing image retrieval and
classification with a new large-scale training and testing dataset, SF300,
including both vertical and oblique aerial images and made available to the
research community, and an associated fine-tuning method. We additionally
propose a new adversarial fine-tuning method for global descriptors. We show
that our framework systematically achieves a boost of retrieval and
classification performance on nine different datasets compared to an ImageNet
pretrained baseline, with currently no other method to compare to.
