  Variational inference is increasingly being addressed with stochastic
optimization. In this setting, the gradient's variance plays a crucial role in
the optimization procedure, since high variance gradients lead to poor
convergence. A popular approach used to reduce gradient's variance involves the
use of control variates. Despite the good results obtained, control variates
developed for variational inference are typically looked at in isolation. In
this paper we clarify the large number of control variates that are available
by giving a systematic view of how they are derived. We also present a Bayesian
risk minimization framework in which the quality of a procedure for combining
control variates is quantified by its effect on optimization convergence rates,
which leads to a very simple combination rule. Results show that combining a
large number of control variates this way significantly improves the
convergence of inference over using the typical gradient estimators or a
reduced number of control variates.
