  Normalizing flow (NF) has gained popularity over traditional maximum
likelihood based methods due to its strong capability to model complex data
distributions. However, the standard approach, which maps the observed data to
a normal distribution, has difficulty in handling data distributions with
multiple relatively isolated modes. To overcome this issue, we propose a new
framework based on variational latent representation to improve the practical
performance of NF. The idea is to replace the standard normal latent variable
with a more general latent representation, jointly learned via Variational
Bayes. For example, by taking the latent representation as a discrete sequence,
our framework can learn a Transformer model that generates the latent sequence
and an NF model that generates continuous data distribution conditioned on the
sequence. The resulting method is significantly more powerful than the standard
normalization flow approach for generating data distributions with multiple
modes. Extensive experiments have shown the advantages of NF with variational
latent representation.
