  This paper investigates the capability of plain Vision Transformers (ViTs)
for semantic segmentation using the encoder-decoder framework and introduces
\textbf{SegViTv2}. In this study, we introduce a novel Attention-to-Mask (\atm)
module to design a lightweight decoder effective for plain ViT. The proposed
ATM converts the global attention map into semantic masks for high-quality
segmentation results. Our decoder outperforms the popular decoder UPerNet using
various ViT backbones while consuming only about $5\%$ of the computational
cost. For the encoder, we address the concern of the relatively high
computational cost in the ViT-based encoders and propose a \emph{Shrunk++}
structure that incorporates edge-aware query-based down-sampling (EQD) and
query-based upsampling (QU) modules. The Shrunk++ structure reduces the
computational cost of the encoder by up to $50\%$ while maintaining competitive
performance. Furthermore, we propose to adapt SegViT for continual semantic
segmentation, demonstrating nearly zero forgetting of previously learned
knowledge. Experiments show that our proposed SegViTv2 surpasses recent
segmentation methods on three popular benchmarks including ADE20k,
COCO-Stuff-10k and PASCAL-Context datasets. The code is available through the
following link: \url{https://github.com/zbwxp/SegVit}.
