  Deep learning on point clouds plays a vital role in a wide range of
applications such as autonomous driving and AR/VR. These applications interact
with people in real-time on edge devices and thus require low latency and low
energy. Compared to projecting the point cloud to 2D space, directly processing
the 3D point cloud yields higher accuracy and lower #MACs. However, the
extremely sparse nature of point cloud poses challenges to hardware
acceleration. For example, we need to explicitly determine the nonzero outputs
and search for the nonzero neighbors (mapping operation), which is unsupported
in existing accelerators. Furthermore, explicit gather and scatter of sparse
features are required, resulting in large data movement overhead. In this
paper, we comprehensively analyze the performance bottleneck of modern point
cloud networks on CPU/GPU/TPU. To address the challenges, we then present
PointAcc, a novel point cloud deep learning accelerator. PointAcc maps diverse
mapping operations onto one versatile ranking-based kernel, streams the sparse
computation with configurable caching, and temporally fuses consecutive dense
layers to reduce the memory footprint. Evaluated on 8 point cloud models across
4 applications, PointAcc achieves 3.7X speedup and 22X energy savings over RTX
2080Ti GPU. Co-designed with light-weight neural networks, PointAcc rivals the
prior accelerator Mesorasi by 100X speedup with 9.1% higher accuracy running
segmentation on the S3DIS dataset. PointAcc paves the way for efficient point
cloud recognition.
