  Vision Transformer (ViT) extracts the final representation from either class
token or an average of all patch tokens, following the architecture of
Transformer in Natural Language Processing (NLP) or Convolutional Neural
Networks (CNNs) in computer vision. However, studies for the best way of
aggregating the patch tokens are still limited to average pooling, while
widely-used pooling strategies, such as max and GeM pooling, can be considered.
Despite their effectiveness, the existing pooling strategies do not consider
the architecture of ViT and the channel-wise difference in the activation maps,
aggregating the crucial and trivial channels with the same importance. In this
paper, we present Group Generalized Mean (GGeM) pooling as a simple yet
powerful pooling strategy for ViT. GGeM divides the channels into groups and
computes GeM pooling with a shared pooling parameter per group. As ViT groups
the channels via a multi-head attention mechanism, grouping the channels by
GGeM leads to lower head-wise dependence while amplifying important channels on
the activation maps. Exploiting GGeM shows 0.1%p to 0.7%p performance boosts
compared to the baselines and achieves state-of-the-art performance for
ViT-Base and ViT-Large models in ImageNet-1K classification task. Moreover,
GGeM outperforms the existing pooling strategies on image retrieval and
multi-modal representation learning tasks, demonstrating the superiority of
GGeM for a variety of tasks. GGeM is a simple algorithm in that only a few
lines of code are necessary for implementation.
