  Recent studies of gradient descent with large step sizes have shown that
there is often a regime with an initial increase in the largest eigenvalue of
the loss Hessian (progressive sharpening), followed by a stabilization of the
eigenvalue near the maximum value which allows convergence (edge of stability).
These phenomena are intrinsically non-linear and do not happen for models in
the constant Neural Tangent Kernel (NTK) regime, for which the predictive
function is approximately linear in the parameters. As such, we consider the
next simplest class of predictive models, namely those that are quadratic in
the parameters, which we call second-order regression models. For quadratic
objectives in two dimensions, we prove that this second-order regression model
exhibits progressive sharpening of the NTK eigenvalue towards a value that
differs slightly from the edge of stability, which we explicitly compute. In
higher dimensions, the model generically shows similar behavior, even without
the specific structure of a neural network, suggesting that progressive
sharpening and edge-of-stability behavior aren't unique features of neural
networks, and could be a more general property of discrete learning algorithms
in high-dimensional non-linear models.
