  The limit of infinite width allows for substantial simplifications in the
analytical study of over-parameterised neural networks. With a suitable random
initialisation, an extremely large network exhibits an approximately Gaussian
behaviour. In the present work, we establish a similar result for a simple
stochastic architecture whose parameters are random variables, holding both
before and during training. The explicit evaluation of the output distribution
allows for a PAC-Bayesian training procedure that directly optimises the
generalisation bound. For a large but finite-width network, we show empirically
on MNIST that this training approach can outperform standard PAC-Bayesian
methods.
