  Video Panoptic Segmentation (VPS) aims to achieve comprehensive pixel-level
scene understanding by segmenting all pixels and associating objects in a
video. Current solutions can be categorized into online and near-online
approaches. Evolving over the time, each category has its own specialized
designs, making it nontrivial to adapt models between different categories. To
alleviate the discrepancy, in this work, we propose a unified approach for
online and near-online VPS. The meta architecture of the proposed Video-kMaX
consists of two components: within clip segmenter (for clip-level segmentation)
and cross-clip associater (for association beyond clips). We propose clip-kMaX
(clip k-means mask transformer) and HiLA-MB (Hierarchical Location-Aware Memory
Buffer) to instantiate the segmenter and associater, respectively. Our general
formulation includes the online scenario as a special case by adopting clip
length of one. Without bells and whistles, Video-kMaX sets a new
state-of-the-art on KITTI-STEP and VIPSeg for video panoptic segmentation, and
VSPW for video semantic segmentation. Code will be made publicly available.
