  Robust real-time detection and motion forecasting of traffic participants is
necessary for autonomous vehicles to safely navigate urban environments. In
this paper, we present RV-FuseNet, a novel end-to-end approach for joint
detection and trajectory estimation directly from time-series LiDAR data.
Instead of the widely used bird's eye view (BEV) representation, we utilize the
native range view (RV) representation of LiDAR data. The RV preserves the full
resolution of the sensor by avoiding the voxelization used in the BEV.
Furthermore, RV can be processed efficiently due to its compactness. Previous
approaches project time-series data to a common viewpoint for temporal fusion,
and often this viewpoint is different from where it was captured. This is
sufficient for BEV methods, but for RV methods, this can lead to loss of
information and data distortion which has an adverse impact on performance. To
address this challenge we propose a simple yet effective novel architecture,
\textit{Incremental Fusion}, that minimizes the information loss by
sequentially projecting each RV sweep into the viewpoint of the next sweep in
time. We show that our approach significantly improves motion forecasting
performance over the existing state-of-the-art. Furthermore, we demonstrate
that our sequential fusion approach is superior to alternative RV based fusion
methods on multiple datasets.
