  LiDAR-based semantic perception tasks are critical yet challenging for
autonomous driving. Due to the motion of objects and static/dynamic occlusion,
temporal information plays an essential role in reinforcing perception by
enhancing and completing single-frame knowledge. Previous approaches either
directly stack historical frames to the current frame or build a 4D
spatio-temporal neighborhood using KNN, which duplicates computation and
hinders realtime performance. Based on our observation that stacking all the
historical points would damage performance due to a large amount of redundant
and misleading information, we propose the Sparse Voxel-Adjacent Query Network
(SVQNet) for 4D LiDAR semantic segmentation. To take full advantage of the
historical frames high-efficiently, we shunt the historical points into two
groups with reference to the current points. One is the Voxel-Adjacent
Neighborhood carrying local enhancing knowledge. The other is the Historical
Context completing the global knowledge. Then we propose new modules to select
and extract the instructive features from the two groups. Our SVQNet achieves
state-of-the-art performance in LiDAR semantic segmentation of the
SemanticKITTI benchmark and the nuScenes dataset.
