  Mean field variational Bayes (MFVB) is a popular posterior approximation
method due to its fast runtime on large-scale data sets. However, it is well
known that a major failing of MFVB is that it underestimates the uncertainty of
model variables (sometimes severely) and provides no information about model
variable covariance.
  We generalize linear response methods from statistical physics to deliver
accurate uncertainty estimates for model variables---both for individual
variables and coherently across variables. We call our method linear response
variational Bayes (LRVB). When the MFVB posterior approximation is in the
exponential family, LRVB has a simple, analytic form, even for non-conjugate
models. Indeed, we make no assumptions about the form of the true posterior. We
demonstrate the accuracy and scalability of our method on a range of models for
both simulated and real data.
