  Variational Inference is a powerful tool in the Bayesian modeling toolkit,
however, its effectiveness is determined by the expressivity of the utilized
variational distributions in terms of their ability to match the true posterior
distribution. In turn, the expressivity of the variational family is largely
limited by the requirement of having a tractable density function. To overcome
this roadblock, we introduce a new family of variational upper bounds on a
marginal log density in the case of hierarchical models (also known as latent
variable models). We then give an upper bound on the Kullback-Leibler
divergence and derive a family of increasingly tighter variational lower bounds
on the otherwise intractable standard evidence lower bound for hierarchical
variational distributions, enabling the use of more expressive approximate
posteriors. We show that previously known methods, such as Hierarchical
Variational Models, Semi-Implicit Variational Inference and Doubly
Semi-Implicit Variational Inference can be seen as special cases of the
proposed approach, and empirically demonstrate superior performance of the
proposed method in a set of experiments.
