  The posterior collapse phenomenon in variational autoencoder (VAE), where the
variational posterior distribution closely matches the prior distribution, can
hinder the quality of the learned latent variables. As a consequence of
posterior collapse, the latent variables extracted by the encoder in VAE
preserve less information from the input data and thus fail to produce
meaningful representations as input to the reconstruction process in the
decoder. While this phenomenon has been an actively addressed topic related to
VAE performance, the theory for posterior collapse remains underdeveloped,
especially beyond the standard VAE. In this work, we advance the theoretical
understanding of posterior collapse to two important and prevalent yet less
studied classes of VAE: conditional VAE and hierarchical VAE. Specifically, via
a non-trivial theoretical analysis of linear conditional VAE and hierarchical
VAE with two levels of latent, we prove that the cause of posterior collapses
in these models includes the correlation between the input and output of the
conditional VAE and the effect of learnable encoder variance in the
hierarchical VAE. We empirically validate our theoretical findings for linear
conditional and hierarchical VAE and demonstrate that these results are also
predictive for non-linear cases with extensive experiments.
