  This paper presents a novel framework to integrate both semantic and instance
contexts for panoptic segmentation. In existing works, it is common to use a
shared backbone to extract features for both things (countable classes such as
vehicles) and stuff (uncountable classes such as roads). This, however, fails
to capture the rich relations among them, which can be utilized to enhance
visual understanding and segmentation performance. To address this shortcoming,
we propose a novel Panoptic, Instance, and Semantic Relations (PISR) module to
exploit such contexts. First, we generate panoptic encodings to summarize key
features of the semantic classes and predicted instances. A Panoptic Relational
Attention (PRA) module is then applied to the encodings and the global feature
map from the backbone. It produces a feature map that captures 1) the relations
across semantic classes and instances and 2) the relations between these
panoptic categories and spatial features. PISR also automatically learns to
focus on the more important instances, making it robust to the number of
instances used in the relational attention module. Moreover, PISR is a general
module that can be applied to any existing panoptic segmentation architecture.
Through extensive evaluations on panoptic segmentation benchmarks like
Cityscapes, COCO, and ADE20K, we show that PISR attains considerable
improvements over existing approaches.
