  As a specific semantic segmentation task, aerial imagery segmentation has
been widely employed in high spatial resolution (HSR) remote sensing images
understanding. Besides common issues (e.g. large scale variation) faced by
general semantic segmentation tasks, aerial imagery segmentation has some
unique challenges, the most critical one among which lies in
foreground-background imbalance. There have been some recent efforts that
attempt to address this issue by proposing sophisticated neural network
architectures, since they can be used to extract informative multi-scale
feature representations and increase the discrimination of object boundaries.
Nevertheless, many of them merely utilize those multi-scale representations in
ad-hoc measures but disregard the fact that the semantic meaning of objects
with various sizes could be better identified via receptive fields of diverse
ranges. In this paper, we propose Adaptive Focus Framework (AF$_2$), which
adopts a hierarchical segmentation procedure and focuses on adaptively
utilizing multi-scale representations generated by widely adopted neural
network architectures. Particularly, a learnable module, called Adaptive
Confidence Mechanism (ACM), is proposed to determine which scale of
representation should be used for the segmentation of different objects.
Comprehensive experiments show that AF$_2$ has significantly improved the
accuracy on three widely used aerial benchmarks, as fast as the mainstream
method.
