  While point-based neural architectures have demonstrated their efficacy, the
time-consuming sampler currently prevents them from performing real-time
reasoning on scene-level point clouds. Existing methods attempt to overcome
this issue by using random sampling strategy instead of the commonly-adopted
farthest point sampling~(FPS), but at the expense of lower performance. So the
effectiveness/efficiency trade-off remains under-explored. In this paper, we
reveal the key to high-quality sampling is ensuring an even spacing between
points in the subset, which can be naturally obtained through a grid. Based on
this insight, we propose a hierarchical adaptive voxel-guided point sampler
with linear complexity and high parallelization for real-time applications.
Extensive experiments on large-scale point cloud detection and segmentation
tasks demonstrate that our method achieves competitive performance with the
most powerful FPS, at an amazing speed that is more than 100 times faster. This
breakthrough in efficiency addresses the bottleneck of the sampling step when
handling scene-level point clouds. Furthermore, our sampler can be easily
integrated into existing models and achieves a 20$\sim$80\% reduction in
runtime with minimal effort. The code will be available at
https://github.com/OuyangJunyuan/pointcloud-3d-detector-tensorrt
