  The recently developed pure Transformer architectures have attained promising
accuracy on point cloud learning benchmarks compared to convolutional neural
networks. However, existing point cloud Transformers are computationally
expensive since they waste a significant amount of time on structuring the
irregular data. To solve this shortcoming, we present Sparse Window Attention
(SWA) module to gather coarse-grained local features from non-empty voxels,
which not only bypasses the expensive irregular data structuring and invalid
empty voxel computation, but also obtains linear computational complexity with
respect to voxel resolution. Meanwhile, to gather fine-grained features about
the global shape, we introduce relative attention (RA) module, a more robust
self-attention variant for rigid transformations of objects. Equipped with the
SWA and RA, we construct our neural architecture called PVT that integrates
both modules into a joint framework for point cloud learning. Compared with
previous Transformer-based and attention-based models, our method attains top
accuracy of 94.0% on classification benchmark and 10x inference speedup on
average. Extensive experiments also valid the effectiveness of PVT on part and
semantic segmentation benchmarks (86.6% and 69.2% mIoU, respectively).
