  Semantic segmentation benchmarks in the realm of autonomous driving are
dominated by large pre-trained transformers, yet their widespread adoption is
impeded by substantial computational costs and prolonged training durations. To
lift this constraint, we look at efficient semantic segmentation from a
perspective of comprehensive knowledge distillation and consider to bridge the
gap between multi-source knowledge extractions and transformer-specific patch
embeddings. We put forward the Transformer-based Knowledge Distillation
(TransKD) framework which learns compact student transformers by distilling
both feature maps and patch embeddings of large teacher transformers, bypassing
the long pre-training process and reducing the FLOPs by >85.0%. Specifically,
we propose two fundamental and two optimization modules: (1) Cross Selective
Fusion (CSF) enables knowledge transfer between cross-stage features via
channel attention and feature map distillation within hierarchical
transformers; (2) Patch Embedding Alignment (PEA) performs dimensional
transformation within the patchifying process to facilitate the patch embedding
distillation; (3) Global-Local Context Mixer (GL-Mixer) extracts both global
and local information of a representative embedding; (4) Embedding Assistant
(EA) acts as an embedding method to seamlessly bridge teacher and student
models with the teacher's number of channels. Experiments on Cityscapes, ACDC,
NYUv2, and Pascal VOC2012 datasets show that TransKD outperforms
state-of-the-art distillation frameworks and rivals the time-consuming
pre-training method. The source code is publicly available at
https://github.com/RuipingL/TransKD.
