  Generally speaking, the main objective when training a neural speech
synthesis system is to synthesize natural and expressive speech from the output
layer of the neural network without much attention given to the hidden layers.
However, by learning useful latent representation, the system can be used for
many more practical scenarios. In this paper, we investigate the use of
quantized vectors to model the latent linguistic embedding and compare it with
the continuous counterpart. By enforcing different policies over the latent
spaces in the training, we are able to obtain a latent linguistic embedding
that takes on different properties while having a similar performance in terms
of quality and speaker similarity. Our experiments show that the voice cloning
system built with vector quantization has only a small degradation in terms of
perceptive evaluations, but has a discrete latent space that is useful for
reducing the representation bit-rate, which is desirable for data transferring,
or limiting the information leaking, which is important for speaker
anonymization and other tasks of that nature.
