  Transformers have proved to be very effective for visual recognition tasks.
In particular, vision transformers construct compressed global representations
through self-attention and learnable class tokens. Multi-resolution
transformers have shown recent successes in semantic segmentation but can only
capture local interactions in high-resolution feature maps. This paper extends
the notion of global tokens to build GLobal Attention Multi-resolution (GLAM)
transformers. GLAM is a generic module that can be integrated into most
existing transformer backbones. GLAM includes learnable global tokens, which
unlike previous methods can model interactions between all image regions, and
extracts powerful representations during training. Extensive experiments show
that GLAM-Swin or GLAM-Swin-UNet exhibit substantially better performances than
their vanilla counterparts on ADE20K and Cityscapes. Moreover, GLAM can be used
to segment large 3D medical images, and GLAM-nnFormer achieves new
state-of-the-art performance on the BCV dataset.
