  Nearest neighbour based methods have proved to be one of the most successful
self-supervised learning (SSL) approaches due to their high generalization
capabilities. However, their computational efficiency decreases when more than
one neighbour is used. In this paper, we propose a novel contrastive SSL
approach, which we call All4One, that reduces the distance between neighbour
representations using ''centroids'' created through a self-attention mechanism.
We use a Centroid Contrasting objective along with single Neighbour Contrasting
and Feature Contrasting objectives. Centroids help in learning contextual
information from multiple neighbours whereas the neighbour contrast enables
learning representations directly from the neighbours and the feature contrast
allows learning representations unique to the features. This combination
enables All4One to outperform popular instance discrimination approaches by
more than 1% on linear classification evaluation for popular benchmark datasets
and obtains state-of-the-art (SoTA) results. Finally, we show that All4One is
robust towards embedding dimensionalities and augmentations, surpassing NNCLR
and Barlow Twins by more than 5% on low dimensionality and weak augmentation
settings. The source code would be made available soon.
