  This work examines the deep disconnect between existing theoretical analyses
of gradient-based algorithms and the practice of training deep neural networks.
Specifically, we provide numerical evidence that in large-scale neural network
training (e.g., ImageNet + ResNet101, and WT103 + TransformerXL models), the
neural network's weights do not converge to stationary points where the
gradient of the loss is zero. Remarkably, however, we observe that even though
the weights do not converge to stationary points, the progress in minimizing
the loss function halts and training loss stabilizes. Inspired by this
observation, we propose a new perspective based on ergodic theory of dynamical
systems to explain it. Rather than studying the evolution of weights, we study
the evolution of the distribution of weights. We prove convergence of the
distribution of weights to an approximate invariant measure, thereby explaining
how the training loss can stabilize without weights necessarily converging to
stationary points. We further discuss how this perspective can better align
optimization theory with empirical observations in machine learning practice.
