  Recently, Transformers have shown promising performance in various vision
tasks. However, the high costs of global self-attention remain challenging for
Transformers, especially for high-resolution vision tasks. Local self-attention
runs attention computation within a limited region for the sake of efficiency,
resulting in insufficient context modeling as their receptive fields are small.
In this work, we introduce two new attention modules to enhance the global
modeling capability of the hierarchical vision transformer, namely, random
sampling windows (RS-Win) and important region windows (IR-Win). Specifically,
RS-Win sample random image patches to compose the window, following a uniform
distribution, i.e., the patches in RS-Win can come from any position in the
image. IR-Win composes the window according to the weights of the image patches
in the attention map. Notably, RS-Win is able to capture global information
throughout the entire model, even in earlier, high-resolution stages. IR-Win
enables the self-attention module to focus on important regions of the image
and capture more informative features. Incorporated with these designs,
RSIR-Win Transformer demonstrates competitive performance on common vision
tasks.
