  The attention mechanism can refine the extracted feature maps and boost the
classification performance of the deep network, which has become an essential
technique in computer vision and natural language processing. However, the
memory and computational costs of the dot-product attention mechanism increase
quadratically with the spatio-temporal size of the input. Such growth hinders
the usage of attention mechanisms considerably in application scenarios with
large-scale inputs. In this Letter, we propose a Linear Attention Mechanism
(LAM) to address this issue, which is approximately equivalent to dot-product
attention with computational efficiency. Such a design makes the incorporation
between attention mechanisms and deep networks much more flexible and
versatile. Based on the proposed LAM, we re-factor the skip connections in the
raw U-Net and design a Multi-stage Attention ResU-Net (MAResU-Net) for semantic
segmentation from fine-resolution remote sensing images. Experiments conducted
on the Vaihingen dataset demonstrated the effectiveness and efficiency of our
MAResU-Net. Open-source code is available at
https://github.com/lironui/Multistage-Attention-ResU-Net.
