  Diagonal linear networks (DLNs) are a toy simplification of artificial neural
networks; they consist in a quadratic reparametrization of linear regression
inducing a sparse implicit regularization. In this paper, we describe the
trajectory of the gradient flow of DLNs in the limit of small initialization.
We show that incremental learning is effectively performed in the limit:
coordinates are successively activated, while the iterate is the minimizer of
the loss constrained to have support on the active coordinates only. This shows
that the sparse implicit regularization of DLNs decreases with time. This work
is restricted to the underparametrized regime with anti-correlated features for
technical reasons.
