  The recognition network in deep latent variable models such as variational
autoencoders (VAEs) relies on amortized inference for efficient posterior
approximation that can scale up to large datasets. However, this technique has
also been demonstrated to select suboptimal variational parameters, often
resulting in considerable additional error called the amortization gap. To
close the amortization gap and improve the training of the generative model,
recent works have introduced an additional refinement step that applies
stochastic variational inference (SVI) to improve upon the variational
parameters returned by the amortized inference model. In this paper, we propose
the Buffered Stochastic Variational Inference (BSVI), a new refinement
procedure that makes use of SVI's sequence of intermediate variational proposal
distributions and their corresponding importance weights to construct a new
generalized importance-weighted lower bound. We demonstrate empirically that
training the variational autoencoders with BSVI consistently out-performs SVI,
yielding an improved training procedure for VAEs.
