  In the wake of Masked Image Modeling (MIM), a diverse range of plain,
non-hierarchical Vision Transformer (ViT) models have been pre-trained with
extensive datasets, offering new paradigms and significant potential for
semantic segmentation. Current state-of-the-art systems incorporate numerous
inductive biases and employ cumbersome decoders. Building upon the original
motivations of plain ViTs, which are simplicity and generality, we explore
high-performance `minimalist' systems to this end. Our primary purpose is to
provide simple and efficient baselines for practical semantic segmentation with
plain ViTs. Specifically, we first explore the feasibility and methodology for
achieving high-performance semantic segmentation using the last feature map. As
a result, we introduce the PlainSeg, a model comprising only three 3$\times$3
convolutions in addition to the transformer layers (either encoder or decoder).
In this process, we offer insights into two underlying principles: (i)
high-resolution features are crucial to high performance in spite of employing
simple up-sampling techniques and (ii) the slim transformer decoder requires a
much larger learning rate than the wide transformer decoder. On this basis, we
further present the PlainSeg-Hier, which allows for the utilization of
hierarchical features. Extensive experiments on four popular benchmarks
demonstrate the high performance and efficiency of our methods. They can also
serve as powerful tools for assessing the transfer ability of base models in
semantic segmentation. Code is available at
\url{https://github.com/ydhongHIT/PlainSeg}.
