  Approximating complex probability densities is a core problem in modern
statistics. In this paper, we introduce the concept of Variational Inference
(VI), a popular method in machine learning that uses optimization techniques to
estimate complex probability densities. This property allows VI to converge
faster than classical methods, such as, Markov Chain Monte Carlo sampling.
Conceptually, VI works by choosing a family of probability density functions
and then finding the one closest to the actual probability density -- often
using the Kullback-Leibler (KL) divergence as the optimization metric. We
introduce the Evidence Lower Bound to tractably compute the approximated
probability density and we review the ideas behind mean-field variational
inference. Finally, we discuss the applications of VI to variational
auto-encoders (VAE) and VAE-Generative Adversarial Network (VAE-GAN). With this
paper, we aim to explain the concept of VI and assist in future research with
this approach.
