  From the sampling of data to the initialisation of parameters, randomness is
ubiquitous in modern Machine Learning practice. Understanding the statistical
fluctuations engendered by the different sources of randomness in prediction is
therefore key to understanding robust generalisation. In this manuscript we
develop a quantitative and rigorous theory for the study of fluctuations in an
ensemble of generalised linear models trained on different, but correlated,
features in high-dimensions. In particular, we provide a complete description
of the asymptotic joint distribution of the empirical risk minimiser for
generic convex loss and regularisation in the high-dimensional limit. Our
result encompasses a rich set of classification and regression tasks, such as
the lazy regime of overparametrised neural networks, or equivalently the random
features approximation of kernels. While allowing to study directly the
mitigating effect of ensembling (or bagging) on the bias-variance decomposition
of the test error, our analysis also helps disentangle the contribution of
statistical fluctuations, and the singular role played by the interpolation
threshold that are at the roots of the "double-descent" phenomenon.
