  We consider the problem of learning a target function corresponding to a
deep, extensive-width, non-linear neural network with random Gaussian weights.
We consider the asymptotic limit where the number of samples, the input
dimension and the network width are proportionally large. We propose a
closed-form expression for the Bayes-optimal test error, for regression and
classification tasks. We further compute closed-form expressions for the test
errors of ridge regression, kernel and random features regression. We find, in
particular, that optimally regularized ridge regression, as well as kernel
regression, achieve Bayes-optimal performances, while the logistic loss yields
a near-optimal test error for classification. We further show numerically that
when the number of samples grows faster than the dimension, ridge and kernel
methods become suboptimal, while neural networks achieve test error close to
zero from quadratically many samples.
