  We introduce PointConvFormer, a novel building block for point cloud based
deep network architectures. Inspired by generalization theory, PointConvFormer
combines ideas from point convolution, where filter weights are only based on
relative position, and Transformers which utilize feature-based attention. In
PointConvFormer, attention computed from feature difference between points in
the neighborhood is used to modify the convolutional weights at each point.
Hence, we preserved the invariances from point convolution, whereas attention
helps to select relevant points in the neighborhood for convolution.
PointConvFormer is suitable for multiple tasks that require details at the
point level, such as segmentation and scene flow estimation tasks. We
experiment on both tasks with multiple datasets including ScanNet,
SemanticKitti, FlyingThings3D and KITTI. Our results show that PointConvFormer
offers a better accuracy-speed tradeoff than classic convolutions, regular
transformers, and voxelized sparse convolution approaches. Visualizations show
that PointConvFormer performs similarly to convolution on flat areas, whereas
the neighborhood selection effect is stronger on object boundaries, showing
that it has got the best of both worlds.
