  Panoptic Part Segmentation (PPS) aims to unify panoptic segmentation and part
segmentation into one task. Previous work mainly utilizes separated approaches
to handle thing, stuff, and part predictions individually without performing
any shared computation and task association. In this work, we aim to unify
these tasks at the architectural level, designing the first end-to-end unified
method named Panoptic-PartFormer. In particular, motivated by the recent
progress in Vision Transformer, we model things, stuff, and part as object
queries and directly learn to optimize the all three predictions as unified
mask prediction and classification problem. We design a decoupled decoder to
generate part feature and thing/stuff feature respectively. Then we propose to
utilize all the queries and corresponding features to perform reasoning jointly
and iteratively. The final mask can be obtained via inner product between
queries and the corresponding features. The extensive ablation studies and
analysis prove the effectiveness of our framework. Our Panoptic-PartFormer
achieves the new state-of-the-art results on both Cityscapes PPS and Pascal
Context PPS datasets with at least 70% GFlops and 50% parameters decrease. In
particular, we get 3.4% relative improvements with ResNet50 backbone and 10%
improvements after adopting Swin Transformer on Pascal Context PPS dataset. To
the best of our knowledge, we are the first to solve the PPS problem via
\textit{a unified and end-to-end transformer model. Given its effectiveness and
conceptual simplicity, we hope our Panoptic-PartFormer can serve as a good
baseline and aid future unified research for PPS. Our code and models are
available at https://github.com/lxtGH/Panoptic-PartFormer.
