  Panoptic segmentation involves a combination of joint semantic segmentation
and instance segmentation, where image contents are divided into two types:
things and stuff. We present Panoptic SegFormer, a general framework for
panoptic segmentation with transformers. It contains three innovative
components: an efficient deeply-supervised mask decoder, a query decoupling
strategy, and an improved post-processing method. We also use Deformable DETR
to efficiently process multi-scale features, which is a fast and efficient
version of DETR. Specifically, we supervise the attention modules in the mask
decoder in a layer-wise manner. This deep supervision strategy lets the
attention modules quickly focus on meaningful semantic regions. It improves
performance and reduces the number of required training epochs by half compared
to Deformable DETR. Our query decoupling strategy decouples the
responsibilities of the query set and avoids mutual interference between things
and stuff. In addition, our post-processing strategy improves performance
without additional costs by jointly considering classification and segmentation
qualities to resolve conflicting mask overlaps. Our approach increases the
accuracy 6.2\% PQ over the baseline DETR model. Panoptic SegFormer achieves
state-of-the-art results on COCO test-dev with 56.2\% PQ. It also shows
stronger zero-shot robustness over existing methods. The code is released at
\url{https://github.com/zhiqi-li/Panoptic-SegFormer}.
