  It is natural to construct a multi-frame instead of a single-frame 3D
detector for a continuous-time stream. Although increasing the number of frames
might improve performance, previous multi-frame studies only used very limited
frames to build their systems due to the dramatically increased computational
and memory cost. To address these issues, we propose a novel on-stream training
and prediction framework that, in theory, can employ an infinite number of
frames while keeping the same amount of computation as a single-frame detector.
This infinite framework (INT), which can be used with most existing detectors,
is utilized, for example, on the popular CenterPoint, with significant latency
reductions and performance improvements. We've also conducted extensive
experiments on two large-scale datasets, nuScenes and Waymo Open Dataset, to
demonstrate the scheme's effectiveness and efficiency. By employing INT on
CenterPoint, we can get around 7% (Waymo) and 15% (nuScenes) performance boost
with only 2~4ms latency overhead, and currently SOTA on the Waymo 3D Detection
leaderboard.
