  We present an Encoder-Decoder Attention Transformer, EDAFormer, which
consists of the Embedding-Free Transformer (EFT) encoder and the all-attention
decoder leveraging our Embedding-Free Attention (EFA) structure. The proposed
EFA is a novel global context modeling mechanism that focuses on functioning
the global non-linearity, not the specific roles of the query, key and value.
For the decoder, we explore the optimized structure for considering the
globality, which can improve the semantic segmentation performance. In
addition, we propose a novel Inference Spatial Reduction (ISR) method for the
computational efficiency. Different from the previous spatial reduction
attention methods, our ISR method further reduces the key-value resolution at
the inference phase, which can mitigate the computation-performance trade-off
gap for the efficient semantic segmentation. Our EDAFormer shows the
state-of-the-art performance with the efficient computation compared to the
existing transformer-based semantic segmentation models in three public
benchmarks, including ADE20K, Cityscapes and COCO-Stuff. Furthermore, our ISR
method reduces the computational cost by up to 61% with minimal mIoU
performance degradation on Cityscapes dataset. The code is available at
https://github.com/hyunwoo137/EDAFormer.
