  Autoregressive language models are powerful and relatively easy to train.
However, these models are usually trained without explicit conditioning labels
and do not offer easy ways to control global aspects such as sentiment or topic
during generation. Bowman & al. (2016) adapted the Variational Autoencoder
(VAE) for natural language with the sequence-to-sequence architecture and
claimed that the latent vector was able to capture such global features in an
unsupervised manner. We question this claim. We measure which words benefit
most from the latent information by decomposing the reconstruction loss per
position in the sentence. Using this method, we find that VAEs are prone to
memorizing the first words and the sentence length, producing local features of
limited usefulness. To alleviate this, we investigate alternative architectures
based on bag-of-words assumptions and language model pretraining. These
variants learn latent variables that are more global, i.e., more predictive of
topic or sentiment labels. Moreover, using reconstructions, we observe that
they decrease memorization: the first word and the sentence length are not
recovered as accurately than with the baselines, consequently yielding more
diverse reconstructions.
