  Variational language models seek to estimate the posterior of latent
variables with an approximated variational posterior. The model often assumes
the variational posterior to be factorized even when the true posterior is not.
The learned variational posterior under this assumption does not capture the
dependency relationships over latent variables. We argue that this would cause
a typical training problem called posterior collapse observed in all other
variational language models. We propose Gaussian Copula Variational Autoencoder
(VAE) to avert this problem. Copula is widely used to model correlation and
dependencies of high-dimensional random variables, and therefore it is helpful
to maintain the dependency relationships that are lost in VAE. The empirical
results show that by modeling the correlation of latent variables explicitly
using a neural parametric copula, we can avert this training difficulty while
getting competitive results among all other VAE approaches.
