  This manuscript considers the problem of learning a random Gaussian network
function using a fully connected network with frozen intermediate layers and
trainable readout layer. This problem can be seen as a natural generalization
of the widely studied random features model to deeper architectures. First, we
prove Gaussian universality of the test error in a ridge regression setting
where the learner and target networks share the same intermediate layers, and
provide a sharp asymptotic formula for it. Establishing this result requires
proving a deterministic equivalent for traces of the deep random features
sample covariance matrices which can be of independent interest. Second, we
conjecture the asymptotic Gaussian universality of the test error in the more
general setting of arbitrary convex losses and generic learner/target
architectures. We provide extensive numerical evidence for this conjecture,
which requires the derivation of closed-form expressions for the layer-wise
post-activation population covariances. In light of our results, we investigate
the interplay between architecture design and implicit regularization.
