  Recent works recognized lidars as an inherently streaming data source and
showed that the end-to-end latency of lidar perception models can be reduced
significantly by operating on wedge-shaped point cloud sectors rather then the
full point cloud. However, due to use of cartesian coordinate systems these
methods represent the sectors as rectangular regions, wasting memory and
compute. In this work we propose using a polar coordinate system and make two
key improvements on this design. First, we increase the spatial context by
using multi-scale padding from neighboring sectors: preceding sector from the
current scan and/or the following sector from the past scan. Second, we improve
the core polar convolutional architecture by introducing feature undistortion
and range stratified convolutions. Experimental results on the nuScenes dataset
show significant improvements over other streaming based methods. We also
achieve comparable results to existing non-streaming methods but with lower
latencies. The code and pretrained models are available at
\url{https://github.com/motional/polarstream}.
