  Non-uniformed 3D sparse data, e.g., point clouds or voxels in different
spatial positions, make contribution to the task of 3D object detection in
different ways. Existing basic components in sparse convolutional networks
(Sparse CNNs) process all sparse data, regardless of regular or submanifold
sparse convolution. In this paper, we introduce two new modules to enhance the
capability of Sparse CNNs, both are based on making feature sparsity learnable
with position-wise importance prediction. They are focal sparse convolution
(Focals Conv) and its multi-modal variant of focal sparse convolution with
fusion, or Focals Conv-F for short. The new modules can readily substitute
their plain counterparts in existing Sparse CNNs and be jointly trained in an
end-to-end fashion. For the first time, we show that spatially learnable
sparsity in sparse convolution is essential for sophisticated 3D object
detection. Extensive experiments on the KITTI, nuScenes and Waymo benchmarks
validate the effectiveness of our approach. Without bells and whistles, our
results outperform all existing single-model entries on the nuScenes test
benchmark at the paper submission time. Code and models are at
https://github.com/dvlab-research/FocalsConv.
