  While the empirical success of self-supervised learning (SSL) heavily relies
on the usage of deep nonlinear models, existing theoretical works on SSL
understanding still focus on linear ones. In this paper, we study the role of
nonlinearity in the training dynamics of contrastive learning (CL) on one and
two-layer nonlinear networks with homogeneous activation $h(x) = h'(x)x$. We
have two major theoretical discoveries. First, the presence of nonlinearity can
lead to many local optima even in 1-layer setting, each corresponding to
certain patterns from the data distribution, while with linear activation, only
one major pattern can be learned. This suggests that models with lots of
parameters can be regarded as a \emph{brute-force} way to find these local
optima induced by nonlinearity. Second, in the 2-layer case, linear activation
is proven not capable of learning specialized weights into diverse patterns,
demonstrating the importance of nonlinearity. In addition, for 2-layer setting,
we also discover \emph{global modulation}: those local patterns discriminative
from the perspective of global-level patterns are prioritized to learn, further
characterizing the learning process. Simulation verifies our theoretical
findings.
