  We present Voxel Transformer (VoTr), a novel and effective voxel-based
Transformer backbone for 3D object detection from point clouds. Conventional 3D
convolutional backbones in voxel-based 3D detectors cannot efficiently capture
large context information, which is crucial for object recognition and
localization, owing to the limited receptive fields. In this paper, we resolve
the problem by introducing a Transformer-based architecture that enables
long-range relationships between voxels by self-attention. Given the fact that
non-empty voxels are naturally sparse but numerous, directly applying standard
Transformer on voxels is non-trivial. To this end, we propose the sparse voxel
module and the submanifold voxel module, which can operate on the empty and
non-empty voxel positions effectively. To further enlarge the attention range
while maintaining comparable computational overhead to the convolutional
counterparts, we propose two attention mechanisms for multi-head attention in
those two modules: Local Attention and Dilated Attention, and we further
propose Fast Voxel Query to accelerate the querying process in multi-head
attention. VoTr contains a series of sparse and submanifold voxel modules and
can be applied in most voxel-based detectors. Our proposed VoTr shows
consistent improvement over the convolutional baselines while maintaining
computational efficiency on the KITTI dataset and the Waymo Open dataset.
