  Depth-aware video panoptic segmentation tackles the inverse projection
problem of restoring panoptic 3D point clouds from video sequences, where the
3D points are augmented with semantic classes and temporally consistent
instance identifiers. We propose a novel solution with a multi-task network
that performs monocular depth estimation and video panoptic segmentation. Since
acquiring ground truth labels for both depth and image segmentation has a
relatively large cost, we leverage the power of unlabeled video sequences with
self-supervised monocular depth estimation and semi-supervised learning from
pseudo-labels for video panoptic segmentation. To further improve the depth
prediction, we introduce panoptic-guided depth losses and a novel panoptic
masking scheme for moving objects to avoid corrupting the training signal.
Extensive experiments on the Cityscapes-DVPS and SemKITTI-DVPS datasets
demonstrate that our model with the proposed improvements achieves competitive
results and fast inference speed.
