  State-of-the-art Variational Auto-Encoders (VAEs) for learning disentangled
latent representations give impressive results in discovering features like
pitch, pause duration, and accent in speech data, leading to highly
controllable text-to-speech (TTS) synthesis. However, these LSTM-based VAEs
fail to learn latent clusters of speaker attributes when trained on either
limited or noisy datasets. Further, different latent variables start encoding
the same features, limiting the control and expressiveness during speech
synthesis. To resolve these issues, we propose RTI-VAE (Reordered Transformer
with Information reduction VAE) where we minimize the mutual information
between different latent variables and devise a modified Transformer
architecture with layer reordering to learn controllable latent representations
in speech data. We show that RTI-VAE reduces the cluster overlap of speaker
attributes by at least 30\% over LSTM-VAE and by at least 7\% over vanilla
Transformer-VAE.
