  The connection between Bayesian neural networks and Gaussian processes gained
a lot of attention in the last few years, with the flagship result that hidden
units converge to a Gaussian process limit when the layers width tends to
infinity. Underpinning this result is the fact that hidden units become
independent in the infinite-width limit. Our aim is to shed some light on
hidden units dependence properties in practical finite-width Bayesian neural
networks. In addition to theoretical results, we assess empirically the depth
and width impacts on hidden units dependence properties.
