  We are living in a three-dimensional space while moving forward through a
fourth dimension: time. To allow artificial intelligence to develop a
comprehensive understanding of such a 4D environment, we introduce 4D Panoptic
Scene Graph (PSG-4D), a new representation that bridges the raw visual data
perceived in a dynamic 4D world and high-level visual understanding.
Specifically, PSG-4D abstracts rich 4D sensory data into nodes, which represent
entities with precise location and status information, and edges, which capture
the temporal relations. To facilitate research in this new area, we build a
richly annotated PSG-4D dataset consisting of 3K RGB-D videos with a total of
1M frames, each of which is labeled with 4D panoptic segmentation masks as well
as fine-grained, dynamic scene graphs. To solve PSG-4D, we propose PSG4DFormer,
a Transformer-based model that can predict panoptic segmentation masks, track
masks along the time axis, and generate the corresponding scene graphs via a
relation component. Extensive experiments on the new dataset show that our
method can serve as a strong baseline for future research on PSG-4D. In the
end, we provide a real-world application example to demonstrate how we can
achieve dynamic scene understanding by integrating a large language model into
our PSG-4D system.
