  We show that the gradient of the cosine similarity between two points goes to
zero in two under-explored settings: (1) if a point has large magnitude or (2)
if the points are on opposite ends of the latent space. Counterintuitively, we
prove that optimizing the cosine similarity between points forces them to grow
in magnitude. Thus, (1) is unavoidable in practice. We then observe that these
derivations are extremely general -- they hold across deep learning
architectures and for many of the standard self-supervised learning (SSL) loss
functions. This leads us to propose cut-initialization: a simple change to
network initialization that helps all studied SSL methods converge faster.
