  We introduce and develop moment propagation for approximate Bayesian
inference. This method can be viewed as a variance correction for mean field
variational Bayes which tends to underestimate posterior variances. Focusing on
the case where the model is described by two sets of parameter vectors, we
develop moment propagation algorithms for linear regression, multivariate
normal, and probit regression models. We show for the probit regression model
that moment propagation empirically performs reasonably well for several
benchmark datasets. Finally, we discuss theoretical gaps and future extensions.
In the supplementary material we show heuristically why moment propagation
leads to appropriate posterior variance estimation, for the linear regression
and multivariate normal models we show precisely why mean field variational
Bayes underestimates certain moments, and prove that our moment propagation
algorithm recovers the exact marginal posterior distributions for all
parameters, and for probit regression we show that moment propagation provides
asymptotically correct posterior means and covariance estimates.
