  We present PVO, a novel panoptic visual odometry framework to achieve more
comprehensive modeling of the scene motion, geometry, and panoptic segmentation
information. Our PVO models visual odometry (VO) and video panoptic
segmentation (VPS) in a unified view, which makes the two tasks mutually
beneficial. Specifically, we introduce a panoptic update module into the VO
Module with the guidance of image panoptic segmentation. This Panoptic-Enhanced
VO Module can alleviate the impact of dynamic objects in the camera pose
estimation with a panoptic-aware dynamic mask. On the other hand, the
VO-Enhanced VPS Module also improves the segmentation accuracy by fusing the
panoptic segmentation result of the current frame on the fly to the adjacent
frames, using geometric information such as camera pose, depth, and optical
flow obtained from the VO Module. These two modules contribute to each other
through recurrent iterative optimization. Extensive experiments demonstrate
that PVO outperforms state-of-the-art methods in both visual odometry and video
panoptic segmentation tasks.
