  Panoptic Part Segmentation (PPS) unifies panoptic and part segmentation into
one task. Previous works utilize separate approaches to handle things, stuff,
and part predictions without shared computation and task association. We aim to
unify these tasks at the architectural level, designing the first end-to-end
unified framework, Panoptic-PartFormer. Moreover, we find the previous metric
PartPQ biases to PQ. To handle both issues, we first design a meta-architecture
that decouples part features and things/stuff features, respectively. We model
things, stuff, and parts as object queries and directly learn to optimize all
three forms of prediction as a unified mask prediction and classification
problem. We term our model as Panoptic-PartFormer. Second, we propose a new
metric Part-Whole Quality (PWQ), better to measure this task from pixel-region
and part-whole perspectives. It also decouples the errors for part segmentation
and panoptic segmentation. Third, inspired by Mask2Former, based on our
meta-architecture, we propose Panoptic-PartFormer++ and design a new part-whole
cross-attention scheme to boost part segmentation qualities further. We design
a new part-whole interaction method using masked cross attention. Finally,
extensive ablation studies and analysis demonstrate the effectiveness of both
Panoptic-PartFormer and Panoptic-PartFormer++. Compared with previous
Panoptic-PartFormer, our Panoptic-PartFormer++ achieves 2% PartPQ and 3% PWQ
improvements on the Cityscapes PPS dataset and 5% PartPQ on the Pascal Context
PPS dataset. On both datasets, Panoptic-PartFormer++ achieves new
state-of-the-art results. Our models can serve as a strong baseline and aid
future research in PPS. The source code and trained models will be available
at~\url{https://github.com/lxtGH/Panoptic-PartFormer}.
