  We propose a black-box variational inference method to approximate
intractable distributions with an increasingly rich approximating class. Our
method, termed variational boosting, iteratively refines an existing
variational approximation by solving a sequence of optimization problems,
allowing the practitioner to trade computation time for accuracy. We show how
to expand the variational approximating class by incorporating additional
covariance structure and by introducing new components to form a mixture. We
apply variational boosting to synthetic and real statistical models, and show
that resulting posterior inferences compare favorably to existing posterior
approximation algorithms in both accuracy and efficiency.
