  Epoch-wise double descent is the phenomenon where generalisation performance
improves beyond the point of overfitting, resulting in a generalisation curve
exhibiting two descents under the course of learning. Understanding the
mechanisms driving this behaviour is crucial not only for understanding the
generalisation behaviour of machine learning models in general, but also for
employing conventional selection methods, such as the use of early stopping to
mitigate overfitting. While we ultimately want to draw conclusions of more
complex models, such as deep neural networks, a majority of theoretical
conclusions regarding the underlying cause of epoch-wise double descent are
based on simple models, such as standard linear regression. To start bridging
this gap, we study epoch-wise double descent in two-layer linear neural
networks. First, we derive a gradient flow for the linear two-layer model, that
bridges the learning dynamics of the standard linear regression model, and the
linear two-layer diagonal network with quadratic weights. Second, we identify
additional factors of epoch-wise double descent emerging with the extra model
layer, by deriving necessary conditions for the generalisation error to follow
a double descent pattern. While epoch-wise double descent in linear regression
has been attributed to differences in input variance, in the two-layer model,
also the singular values of the input-output covariance matrix play an
important role. This opens up for further questions regarding unidentified
factors of epoch-wise double descent for truly deep models.
