  A recent line of work has shown remarkable behaviors of the generalization
error curves in simple learning models. Even the least-squares regression has
shown atypical features such as the model-wise double descent, and further
works have observed triple or multiple descents. Another important
characteristic are the epoch-wise descent structures which emerge during
training. The observations of model-wise and epoch-wise descents have been
analytically derived in limited theoretical settings (such as the random
feature model) and are otherwise experimental. In this work, we provide a full
and unified analysis of the whole time-evolution of the generalization curve,
in the asymptotic large-dimensional regime and under gradient-flow, within a
wider theoretical setting stemming from a gaussian covariate model. In
particular, we cover most cases already disparately observed in the literature,
and also provide examples of the existence of multiple descent structures as a
function of a model parameter or time. Furthermore, we show that our
theoretical predictions adequately match the learning curves obtained by
gradient descent over realistic datasets. Technically we compute averages of
rational expressions involving random matrices using recent developments in
random matrix theory based on "linear pencils". Another contribution, which is
also of independent interest in random matrix theory, is a new derivation of
related fixed point equations (and an extension there-off) using Dyson brownian
motions.
