  Classical learning theory suggests that the optimal generalization
performance of a machine learning model should occur at an intermediate model
complexity, with simpler models exhibiting high bias and more complex models
exhibiting high variance of the predictive function. However, such a simple
trade-off does not adequately describe deep learning models that simultaneously
attain low bias and variance in the heavily overparameterized regime. A primary
obstacle in explaining this behavior is that deep learning algorithms typically
involve multiple sources of randomness whose individual contributions are not
visible in the total variance. To enable fine-grained analysis, we describe an
interpretable, symmetric decomposition of the variance into terms associated
with the randomness from sampling, initialization, and the labels. Moreover, we
compute the high-dimensional asymptotic behavior of this decomposition for
random feature kernel regression, and analyze the strikingly rich phenomenology
that arises. We find that the bias decreases monotonically with the network
width, but the variance terms exhibit non-monotonic behavior and can diverge at
the interpolation boundary, even in the absence of label noise. The divergence
is caused by the \emph{interaction} between sampling and initialization and can
therefore be eliminated by marginalizing over samples (i.e. bagging) \emph{or}
over the initial parameters (i.e. ensemble learning).
