  Prevention of complete and dimensional collapse of representations has
recently become a design principle for self-supervised learning (SSL). However,
questions remain in our theoretical understanding: When do those collapses
occur? What are the mechanisms and causes? We answer these questions by
deriving and thoroughly analyzing an analytically tractable theory of SSL loss
landscapes. In this theory, we identify the causes of the dimensional collapse
and study the effect of normalization and bias. Finally, we leverage the
interpretability afforded by the analytical theory to understand how
dimensional collapse can be beneficial and what affects the robustness of SSL
against data imbalance.
