  This is a tutorial and survey paper on factor analysis, probabilistic
Principal Component Analysis (PCA), variational inference, and Variational
Autoencoder (VAE). These methods, which are tightly related, are dimensionality
reduction and generative models. They assume that every data point is generated
from or caused by a low-dimensional latent factor. By learning the parameters
of distribution of latent space, the corresponding low-dimensional factors are
found for the sake of dimensionality reduction. For their stochastic and
generative behaviour, these models can also be used for generation of new data
points in the data space. In this paper, we first start with variational
inference where we derive the Evidence Lower Bound (ELBO) and Expectation
Maximization (EM) for learning the parameters. Then, we introduce factor
analysis, derive its joint and marginal distributions, and work out its EM
steps. Probabilistic PCA is then explained, as a special case of factor
analysis, and its closed-form solutions are derived. Finally, VAE is explained
where the encoder, decoder and sampling from the latent space are introduced.
Training VAE using both EM and backpropagation are explained.
