  Contrastive learning (CL) has emerged as a powerful technique for
representation learning, with or without label supervision. However, supervised
CL is prone to collapsing representations of subclasses within a class by not
capturing all their features, and unsupervised CL may suppress harder
class-relevant features by focusing on learning easy class-irrelevant features;
both significantly compromise representation quality. Yet, there is no
theoretical understanding of \textit{class collapse} or \textit{feature
suppression} at \textit{test} time. We provide the first unified theoretically
rigorous framework to determine \textit{which} features are learnt by CL. Our
analysis indicate that, perhaps surprisingly, bias of (stochastic) gradient
descent towards finding simpler solutions is a key factor in collapsing
subclass representations and suppressing harder class-relevant features.
Moreover, we present increasing embedding dimensionality and improving the
quality of data augmentations as two theoretically motivated solutions to
{feature suppression}. We also provide the first theoretical explanation for
why employing supervised and unsupervised CL together yields higher-quality
representations, even when using commonly-used stochastic gradient methods.
