  Existing research addresses scene graph generation (SGG) -- a critical
technology for scene understanding in images -- from a detection perspective,
i.e., objects are detected using bounding boxes followed by prediction of their
pairwise relationships. We argue that such a paradigm causes several problems
that impede the progress of the field. For instance, bounding box-based labels
in current datasets usually contain redundant classes like hairs, and leave out
background information that is crucial to the understanding of context. In this
work, we introduce panoptic scene graph generation (PSG), a new problem task
that requires the model to generate a more comprehensive scene graph
representation based on panoptic segmentations rather than rigid bounding
boxes. A high-quality PSG dataset, which contains 49k well-annotated
overlapping images from COCO and Visual Genome, is created for the community to
keep track of its progress. For benchmarking, we build four two-stage
baselines, which are modified from classic methods in SGG, and two one-stage
baselines called PSGTR and PSGFormer, which are based on the efficient
Transformer-based detector, i.e., DETR. While PSGTR uses a set of queries to
directly learn triplets, PSGFormer separately models the objects and relations
in the form of queries from two Transformer decoders, followed by a
prompting-like relation-object matching mechanism. In the end, we share
insights on open challenges and future directions.
