  Despite the widespread adoption of neural networks, their training dynamics
remain poorly understood. We show experimentally that as the size of the
dataset increases, a point forms where the magnitude of the gradient of the
loss becomes unbounded. Gradient descent rapidly brings the network close to
this singularity in parameter space, and further training takes place near it.
This singularity explains a variety of phenomena recently observed in the
Hessian of neural network loss functions, such as training on the edge of
stability and the concentration of the gradient in a top subspace. Once the
network approaches the singularity, the top subspace contributes little to
learning, even though it constitutes the majority of the gradient.
