  Flexible variational distributions improve variational inference but are
harder to optimize. In this work we present a control variate that is
applicable for any reparameterizable distribution with known mean and
covariance matrix, e.g. Gaussians with any covariance structure. The control
variate is based on a quadratic approximation of the model, and its parameters
are set using a double-descent scheme by minimizing the gradient estimator's
variance. We empirically show that this control variate leads to large
improvements in gradient variance and optimization convergence for inference
with non-factorized variational distributions.
