  A lot of recent advances in unsupervised feature learning are based on
designing features which are invariant under semantic data augmentations. A
common way to do this is contrastive learning, which uses positive and negative
samples. Some recent works however have shown promising results for
non-contrastive learning, which does not require negative samples. However, the
non-contrastive losses have obvious "collapsed" minima, in which the encoders
output a constant feature embedding, independent of the input. A folk
conjecture is that so long as these collapsed solutions are avoided, the
produced feature representations should be good. In our paper, we cast doubt on
this story: we show through theoretical results and controlled experiments that
even on simple data models, non-contrastive losses have a preponderance of
non-collapsed bad minima. Moreover, we show that the training process does not
avoid these minima.
