  Variational Autoencoder (VAE) is widely used as a generative model to
approximate a model's posterior on latent variables by combining the amortized
variational inference and deep neural networks. However, when paired with
strong autoregressive decoders, VAE often converges to a degenerated local
optimum known as "posterior collapse". Previous approaches consider the
Kullback Leibler divergence (KL) individual for each datapoint. We propose to
let the KL follow a distribution across the whole dataset, and analyze that it
is sufficient to prevent posterior collapse by keeping the expectation of the
KL's distribution positive. Then we propose Batch Normalized-VAE (BN-VAE), a
simple but effective approach to set a lower bound of the expectation by
regularizing the distribution of the approximate posterior's parameters.
Without introducing any new model component or modifying the objective, our
approach can avoid the posterior collapse effectively and efficiently. We
further show that the proposed BN-VAE can be extended to conditional VAE
(CVAE). Empirically, our approach surpasses strong autoregressive baselines on
language modeling, text classification and dialogue generation, and rivals more
complex approaches while keeping almost the same training time as VAE.
