  Contrastive self-supervised learning (CSL) with a prototypical regularization
has been introduced in learning meaningful representations for downstream tasks
that require strong semantic information. However, to optimize CSL with a loss
that performs the prototypical regularization aggressively, e.g., the ProtoNCE
loss, might cause the "coagulation" of examples in the embedding space. That
is, the intra-prototype diversity of samples collapses to trivial solutions for
their prototype being well-separated from others. Motivated by previous works,
we propose to mitigate this phenomenon by learning Prototypical representation
through Alignment, Uniformity and Correlation (PAUC). Specifically, the
ordinary ProtoNCE loss is revised with: (1) an alignment loss that pulls
embeddings from positive prototypes together; (2) a uniformity loss that
distributes the prototypical level features uniformly; (3) a correlation loss
that increases the diversity and discriminability between prototypical level
features. We conduct extensive experiments on various benchmarks where the
results demonstrate the effectiveness of our method in improving the quality of
prototypical contrastive representations. Particularly, in the classification
down-stream tasks with linear probes, our proposed method outperforms the
state-of-the-art instance-wise and prototypical contrastive learning methods on
the ImageNet-100 dataset by 2.96% and the ImageNet-1K dataset by 2.46% under
the same settings of batch size and epochs.
