  Learning from set-structured data, such as point clouds, has gained
significant attention from the machine learning community. Geometric deep
learning provides a blueprint for designing effective set neural networks that
preserve the permutation symmetry of set-structured data. Of our interest are
permutation invariant networks, which are composed of a permutation equivariant
backbone, permutation invariant global pooling, and regression/classification
head. While existing literature has focused on improving equivariant backbones,
the impact of the pooling layer is often overlooked. In this paper, we examine
the interplay between permutation equivariant backbones and permutation
invariant global pooling on three benchmark point cloud classification
datasets. Our findings reveal that: 1) complex pooling methods, such as
transport-based or attention-based poolings, can significantly boost the
performance of simple backbones, but the benefits diminish for more complex
backbones, 2) even complex backbones can benefit from pooling layers in low
data scenarios, 3) surprisingly, the choice of pooling layers can have a more
significant impact on the model's performance than adjusting the width and
depth of the backbone, and 4) pairwise combination of pooling layers can
significantly improve the performance of a fixed backbone. Our comprehensive
study provides insights for practitioners to design better permutation
invariant set neural networks. Our code is available at
https://github.com/mint-vu/backbone_vs_pooling.
