  Convolutional neural networks (CNNs) and vision transformers (ViTs) have
achieved remarkable success in various vision tasks. However, many
architectures do not consider interactions between feature maps from different
stages and scales, which may limit their performance. In this work, we propose
a simple add-on attention module to overcome these limitations via multi-stage
and cross-scale interactions. Specifically, the proposed Multi-Stage
Cross-Scale Attention (MSCSA) module takes feature maps from different stages
to enable multi-stage interactions and achieves cross-scale interactions by
computing self-attention at different scales based on the multi-stage feature
maps. Our experiments on several downstream tasks show that MSCSA provides a
significant performance boost with modest additional FLOPs and runtime.
