  In nature, symmetry governs regularities, while symmetry breaking brings
texture. In artificial neural networks, symmetry has been a central design
principle to efficiently capture regularities in the world, but the role of
symmetry breaking is not well understood. Here, we develop a theoretical
framework to study the "geometry of learning dynamics" in neural networks, and
reveal a key mechanism of explicit symmetry breaking behind the efficiency and
stability of modern neural networks. To build this understanding, we model the
discrete learning dynamics of gradient descent using a continuous-time
Lagrangian formulation, in which the learning rule corresponds to the kinetic
energy and the loss function corresponds to the potential energy. Then, we
identify "kinetic symmetry breaking" (KSB), the condition when the kinetic
energy explicitly breaks the symmetry of the potential function. We generalize
Noether's theorem known in physics to take into account KSB and derive the
resulting motion of the Noether charge: "Noether's Learning Dynamics" (NLD).
Finally, we apply NLD to neural networks with normalization layers and reveal
how KSB introduces a mechanism of "implicit adaptive optimization",
establishing an analogy between learning dynamics induced by normalization
layers and RMSProp. Overall, through the lens of Lagrangian mechanics, we have
established a theoretical foundation to discover geometric design principles
for the learning dynamics of neural networks.
