  Due to common architecture designs, symmetries exist extensively in
contemporary neural networks. In this work, we unveil the importance of the
loss function symmetries in affecting, if not deciding, the learning behavior
of machine learning models. We prove that every mirror-reflection symmetry,
with reflection surface $O$, in the loss function leads to the emergence of a
constraint on the model parameters $\theta$: $O^T\theta =0$. This constrained
solution becomes satisfied when either the weight decay or gradient noise is
large. Common instances of mirror symmetries in deep learning include
rescaling, rotation, and permutation symmetry. As direct corollaries, we show
that rescaling symmetry leads to sparsity, rotation symmetry leads to low
rankness, and permutation symmetry leads to homogeneous ensembling. Then, we
show that the theoretical framework can explain intriguing phenomena, such as
the loss of plasticity and various collapse phenomena in neural networks, and
suggest how symmetries can be used to design an elegant algorithm to enforce
hard constraints in a differentiable way.
