  Real-time semantic video segmentation is a challenging task due to the strict
requirements of inference speed. Recent approaches mainly devote great efforts
to reducing the model size for high efficiency. In this paper, we rethink this
problem from a different viewpoint: using knowledge contained in compressed
videos. We propose a simple and effective framework, dubbed TapLab, to tap into
resources from the compressed domain. Specifically, we design a fast feature
warping module using motion vectors for acceleration. To reduce the noise
introduced by motion vectors, we design a residual-guided correction module and
a residual-guided frame selection module using residuals. TapLab significantly
reduces redundant computations of the state-of-the-art fast semantic image
segmentation models, running 3 to 10 times faster with controllable accuracy
degradation. The experimental results show that TapLab achieves 70.6% mIoU on
the Cityscapes dataset at 99.8 FPS with a single GPU card for the 1024x2048
videos. A high-speed version even reaches the speed of 160+ FPS. Codes will be
available soon at https://github.com/Sixkplus/TapLab.
