  Characterizing how neural network depth, width, and dataset size jointly
impact model quality is a central problem in deep learning theory. We give here
a complete solution in the special case of linear networks with output
dimension one trained using zero noise Bayesian inference with Gaussian weight
priors and mean squared error as a negative log-likelihood. For any training
dataset, network depth, and hidden layer widths, we find non-asymptotic
expressions for the predictive posterior and Bayesian model evidence in terms
of Meijer-G functions, a class of meromorphic special functions of a single
complex variable. Through novel asymptotic expansions of these Meijer-G
functions, a rich new picture of the joint role of depth, width, and dataset
size emerges. We show that linear networks make provably optimal predictions at
infinite depth: the posterior of infinitely deep linear networks with
data-agnostic priors is the same as that of shallow networks with
evidence-maximizing data-dependent priors. This yields a principled reason to
prefer deeper networks when priors are forced to be data-agnostic. Moreover, we
show that with data-agnostic priors, Bayesian model evidence in wide linear
networks is maximized at infinite depth, elucidating the salutary role of
increased depth for model selection. Underpinning our results is a novel
emergent notion of effective depth, given by the number of hidden layers times
the number of data points divided by the network width; this determines the
structure of the posterior in the large-data limit.
