  Recent work has shown that the prior over functions induced by a deep
Bayesian neural network (BNN) behaves as a Gaussian process (GP) as the width
of all layers becomes large. However, many BNN applications are concerned with
the BNN function space posterior. While some empirical evidence of the
posterior convergence was provided in the original works of Neal (1996) and
Matthews et al. (2018), it is limited to small datasets or architectures due to
the notorious difficulty of obtaining and verifying exactness of BNN posterior
approximations. We provide the missing theoretical proof that the exact BNN
posterior converges (weakly) to the one induced by the GP limit of the prior.
For empirical validation, we show how to generate exact samples from a finite
BNN on a small dataset via rejection sampling.
