  Recently normalizing flows have been gaining traction in text-to-speech (TTS)
and voice conversion (VC) due to their state-of-the-art (SOTA) performance.
Normalizing flows are unsupervised generative models. In this paper, we
introduce supervision to the training process of normalizing flows, without the
need for parallel data. We call this training paradigm AutoEncoder Normalizing
Flow (AE-Flow). It adds a reconstruction loss forcing the model to use
information from the conditioning to reconstruct an audio sample. Our goal is
to understand the impact of each component and find the right combination of
the negative log-likelihood (NLL) and the reconstruction loss in training
normalizing flows with coupling blocks. For that reason we will compare
flow-based mapping model trained with: (i) NLL loss, (ii) NLL and
reconstruction losses, as well as (iii) reconstruction loss only. Additionally,
we compare our model with SOTA VC baseline. The models are evaluated in terms
of naturalness, speaker similarity, intelligibility in many-to-many and
many-to-any VC settings. The results show that the proposed training paradigm
systematically improves speaker similarity and naturalness when compared to
regular training methods of normalizing flows. Furthermore, we show that our
method improves speaker similarity and intelligibility over the
state-of-the-art.
