  The largest eigenvalue of the Hessian, or sharpness, of neural networks is a
key quantity to understand their optimization dynamics. In this paper, we study
the sharpness of deep linear networks for overdetermined univariate regression.
Minimizers can have arbitrarily large sharpness, but not an arbitrarily small
one. Indeed, we show a lower bound on the sharpness of minimizers, which grows
linearly with depth. We then study the properties of the minimizer found by
gradient flow, which is the limit of gradient descent with vanishing learning
rate. We show an implicit regularization towards flat minima: the sharpness of
the minimizer is no more than a constant times the lower bound. The constant
depends on the condition number of the data covariance matrix, but not on width
or depth. This result is proven both for a small-scale initialization and a
residual initialization. Results of independent interest are shown in both
cases. For small-scale initialization, we show that the learned weight matrices
are approximately rank-one and that their singular vectors align. For residual
initialization, convergence of the gradient flow for a Gaussian initialization
of the residual network is proven. Numerical experiments illustrate our results
and connect them to gradient descent with non-vanishing learning rate.
