  Uniformity plays an important role in evaluating learned representations,
providing insights into self-supervised learning. In our quest for effective
uniformity metrics, we pinpoint four principled properties that such metrics
should possess. Namely, an effective uniformity metric should remain invariant
to instance permutations and sample replications while accurately capturing
feature redundancy and dimensional collapse. Surprisingly, we find that the
uniformity metric proposed by \citet{Wang2020UnderstandingCR} fails to satisfy
the majority of these properties. Specifically, their metric is sensitive to
sample replications, and can not account for feature redundancy and dimensional
collapse correctly. To overcome these limitations, we introduce a new
uniformity metric based on the Wasserstein distance, which satisfies all the
aforementioned properties. Integrating this new metric in existing
self-supervised learning methods effectively mitigates dimensional collapse and
consistently improves their performance on downstream tasks involving CIFAR-10
and CIFAR-100 datasets. Code is available at
\url{https://github.com/statsle/WassersteinSSL}.
