  Existing LiDAR-based 3D object detectors usually focus on the single-frame
detection, while ignoring the spatiotemporal information in consecutive point
cloud frames. In this paper, we propose an end-to-end online 3D video object
detector that operates on point cloud sequences. The proposed model comprises a
spatial feature encoding component and a spatiotemporal feature aggregation
component. In the former component, a novel Pillar Message Passing Network
(PMPNet) is proposed to encode each discrete point cloud frame. It adaptively
collects information for a pillar node from its neighbors by iterative message
passing, which effectively enlarges the receptive field of the pillar feature.
In the latter component, we propose an Attentive Spatiotemporal Transformer GRU
(AST-GRU) to aggregate the spatiotemporal information, which enhances the
conventional ConvGRU with an attentive memory gating mechanism. AST-GRU
contains a Spatial Transformer Attention (STA) module and a Temporal
Transformer Attention (TTA) module, which can emphasize the foreground objects
and align the dynamic objects, respectively. Experimental results demonstrate
that the proposed 3D video object detector achieves state-of-the-art
performance on the large-scale nuScenes benchmark.
