  Inference methods are often formulated as variational approximations: these
approximations allow easy evaluation of statistics by marginalization or linear
response, but these estimates can be inconsistent. We show that by introducing
constraints on covariance, one can ensure consistency of linear response with
the variational parameters, and in so doing inference of marginal probability
distributions is improved. For the Bethe approximation and its generalizations,
improvements are achieved with simple choices of the constraints. The
approximations are presented as variational frameworks; iterative procedures
related to message passing are provided for finding the minima.
