  Autonomous robot systems have attracted increasing research attention in
recent years, where environment understanding is a crucial step for robot
navigation, human-robot interaction, and decision. Real-world robot systems
usually collect visual data from multiple sensors and are required to recognize
numerous objects and their movements in complex human-crowded settings.
Traditional benchmarks, with their reliance on single sensors and limited
object classes and scenarios, fail to provide the comprehensive environmental
understanding robots need for accurate navigation, interaction, and
decision-making. As an extension of JRDB dataset, we unveil JRDB-PanoTrack, a
novel open-world panoptic segmentation and tracking benchmark, towards more
comprehensive environmental perception. JRDB-PanoTrack includes (1) various
data involving indoor and outdoor crowded scenes, as well as comprehensive 2D
and 3D synchronized data modalities; (2) high-quality 2D spatial panoptic
segmentation and temporal tracking annotations, with additional 3D label
projections for further spatial understanding; (3) diverse object classes for
closed- and open-world recognition benchmarks, with OSPA-based metrics for
evaluation. Extensive evaluation of leading methods shows significant
challenges posed by our dataset.
