  Empirical and theoretical works show that the input weights of two-layer
neural networks, when initialized with small values, converge towards isolated
orientations. This phenomenon, referred to as condensation, indicates that the
gradient descent methods tend to spontaneously reduce the complexity of neural
networks during the training process. In this work, we elucidate the mechanisms
behind the condensation phenomena occurring in the training of three-layer
neural networks and distinguish it from the training of two-layer neural
networks. Through rigorous theoretical analysis, we establish the blow-up
property of effective dynamics and present a sufficient condition for the
occurrence of condensation, findings that are substantiated by experimental
results. Additionally, we explore the association between condensation and the
low-rank bias observed in deep matrix factorization.
