  There is evidence that transformers offer state-of-the-art recognition
performance on tasks involving overhead imagery (e.g., satellite imagery).
However, it is difficult to make unbiased empirical comparisons between
competing deep learning models, making it unclear whether, and to what extent,
transformer-based models are beneficial. In this paper we systematically
compare the impact of adding transformer structures into state-of-the-art
segmentation models for overhead imagery. Each model is given a similar budget
of free parameters, and their hyperparameters are optimized using Bayesian
Optimization with a fixed quantity of data and computation time. We conduct our
experiments with a large and diverse dataset comprising two large public
benchmarks: Inria and DeepGlobe. We perform additional ablation studies to
explore the impact of specific transformer-based modeling choices. Our results
suggest that transformers provide consistent, but modest, performance
improvements. We only observe this advantage however in hybrid models that
combine convolutional and transformer-based structures, while fully
transformer-based models achieve relatively poor performance.
