  Inference networks of traditional Variational Autoencoders (VAEs) are
typically amortized, resulting in relatively inaccurate posterior approximation
compared to instance-wise variational optimization. Recent semi-amortized
approaches were proposed to address this drawback; however, their iterative
gradient update procedures can be computationally demanding. To address these
issues, in this paper we introduce an accurate amortized inference algorithm.
We propose a novel recursive mixture estimation algorithm for VAEs that
iteratively augments the current mixture with new components so as to maximally
reduce the divergence between the variational and the true posteriors. Using
the functional gradient approach, we devise an intuitive learning criteria for
selecting a new mixture component: the new component has to improve the data
likelihood (lower bound) and, at the same time, be as divergent from the
current mixture distribution as possible, thus increasing representational
diversity. Compared to recently proposed boosted variational inference (BVI),
our method relies on amortized inference in contrast to BVI's non-amortized
single optimization instance. A crucial benefit of our approach is that the
inference at test time requires a single feed-forward pass through the mixture
inference network, making it significantly faster than the semi-amortized
approaches. We show that our approach yields higher test data likelihood than
the state-of-the-art on several benchmark datasets.
