  Learning representations with self-supervision for convolutional networks
(CNN) has been validated to be effective for vision tasks. As an alternative to
CNN, vision transformers (ViT) have strong representation ability with spatial
self-attention and channel-level feedforward networks. Recent works reveal that
self-supervised learning helps unleash the great potential of ViT. Still, most
works follow self-supervised strategies designed for CNN, e.g., instance-level
discrimination of samples, but they ignore the properties of ViT. We observe
that relational modeling on spatial and channel dimensions distinguishes ViT
from other networks. To enforce this property, we explore the feature
SElf-RElation (SERE) for training self-supervised ViT. Specifically, instead of
conducting self-supervised learning solely on feature embeddings from multiple
views, we utilize the feature self-relations, i.e., spatial/channel
self-relations, for self-supervised learning. Self-relation based learning
further enhances the relation modeling ability of ViT, resulting in stronger
representations that stably improve performance on multiple downstream tasks.
Our source code is publicly available at: https://github.com/MCG-NKU/SERE.
