  Efficient representation of point clouds is fundamental for LiDAR-based 3D
object detection. While recent grid-based detectors often encode point clouds
into either voxels or pillars, the distinctions between these approaches remain
underexplored. In this paper, we quantify the differences between the current
encoding paradigms and highlight the limited vertical learning within. To
tackle these limitations, we introduce a hybrid Voxel-Pillar Fusion network
(VPF), which synergistically combines the unique strengths of both voxels and
pillars. Specifically, we first develop a sparse voxel-pillar encoder that
encodes point clouds into voxel and pillar features through 3D and 2D sparse
convolutions respectively, and then introduce the Sparse Fusion Layer (SFL),
facilitating bidirectional interaction between sparse voxel and pillar
features. Our efficient, fully sparse method can be seamlessly integrated into
both dense and sparse detectors. Leveraging this powerful yet straightforward
framework, VPF delivers competitive performance, achieving real-time inference
speeds on the nuScenes and Waymo Open Dataset. The code will be available.
