  Effectively encoding multi-scale contextual information is crucial for
accurate semantic segmentation. Existing transformer-based segmentation models
combine features across scales without any selection, where features on
sub-optimal scales may degrade segmentation outcomes. Leveraging from the
inherent properties of Vision Transformers, we propose a simple yet effective
module, Transformer Scale Gate (TSG), to optimally combine multi-scale
features.TSG exploits cues in self and cross attentions in Vision Transformers
for the scale selection. TSG is a highly flexible plug-and-play module, and can
easily be incorporated with any encoder-decoder-based hierarchical vision
Transformer architecture. Extensive experiments on the Pascal Context and
ADE20K datasets demonstrate that our feature selection strategy achieves
consistent gains.
