  Variational Bayesian Inference is a popular methodology for approximating
posterior distributions over Bayesian neural network weights. Recent work
developing this class of methods has explored ever richer parameterizations of
the approximate posterior in the hope of improving performance. In contrast,
here we share a curious experimental finding that suggests instead restricting
the variational distribution to a more compact parameterization. For a variety
of deep Bayesian neural networks trained using Gaussian mean-field variational
inference, we find that the posterior standard deviations consistently exhibit
strong low-rank structure after convergence. This means that by decomposing
these variational parameters into a low-rank factorization, we can make our
variational approximation more compact without decreasing the models'
performance. Furthermore, we find that such factorized parameterizations
improve the signal-to-noise ratio of stochastic gradient estimates of the
variational lower bound, resulting in faster convergence.
