  The variational autoencoder (VAE) framework remains a popular option for
training unsupervised generative models, especially for discrete data where
generative adversarial networks (GANs) require workaround to create gradient
for the generator. In our work modeling US postal addresses, we show that our
discrete VAE with tree recursive architecture demonstrates limited capability
of capturing field correlations within structured data, even after overcoming
the challenge of posterior collapse with scheduled sampling and tuning of the
KL-divergence weight $\beta$. Worse, VAE seems to have difficulty mapping its
generated samples to the latent space, as their VAE loss lags behind or even
increases during the training process. Motivated by this observation, we show
that augmenting training data with generated variants (augmented training) and
training a VAE with multiple values of $\beta$ simultaneously (multiscale VAE)
both improve the generation quality of VAE. Despite their differences in
motivation and emphasis, we show that augmented training and multiscale VAE are
actually connected and have similar effects on the model.
