  Variational Bayesian posterior inference often requires simplifying
approximations such as mean-field parametrisation to ensure tractability.
However, prior work has associated the variational mean-field approximation for
Bayesian neural networks with underfitting in the case of small datasets or
large model sizes. In this work, we show that invariances in the likelihood
function of over-parametrised models contribute to this phenomenon because
these invariances complicate the structure of the posterior by introducing
discrete and/or continuous modes which cannot be well approximated by Gaussian
mean-field distributions. In particular, we show that the mean-field
approximation has an additional gap in the evidence lower bound compared to a
purpose-built posterior that takes into account the known invariances.
Importantly, this invariance gap is not constant; it vanishes as the
approximation reverts to the prior. We proceed by first considering translation
invariances in a linear model with a single data point in detail. We show that,
while the true posterior can be constructed from a mean-field parametrisation,
this is achieved only if the objective function takes into account the
invariance gap. Then, we transfer our analysis of the linear model to neural
networks. Our analysis provides a framework for future work to explore
solutions to the invariance problem.
