  Self-supervised learning (SSL) has recently achieved significant success in
downstream visual tasks. However, a notable gap still exists between SSL and
supervised learning (SL), especially in complex downstream tasks. In this
paper, we show that the features learned by SSL methods suffer from the
crowding problem, where features of different classes are not distinctly
separated, and features within the same class exhibit large intra-class
variance. In contrast, SL ensures a clear separation between classes. We
analyze this phenomenon and conclude that SSL objectives do not constrain the
relationships between different samples and their augmentations. Our
theoretical analysis delves into how SSL objectives fail to enforce the
necessary constraints between samples and their augmentations, leading to poor
performance in complex tasks. We provide a theoretical framework showing that
the performance gap between SSL and SL mainly stems from the inability of SSL
methods to capture the aggregation of similar augmentations and the separation
of dissimilar augmentations. To address this issue, we propose a learnable
regulator called Dynamic Semantic Adjuster (DSA). DSA aggregates and separates
samples in the feature space while being robust to outliers. Through extensive
empirical evaluations on multiple benchmark datasets, we demonstrate the
superiority of DSA in enhancing feature aggregation and separation, ultimately
closing the performance gap between SSL and SL.
