  We propose a novel training algorithm for a multi-speaker neural
text-to-speech (TTS) model based on multi-task adversarial training. A
conventional generative adversarial network (GAN)-based training algorithm
significantly improves the quality of synthetic speech by reducing the
statistical difference between natural and synthetic speech. However, the
algorithm does not guarantee the generalization performance of the trained TTS
model in synthesizing voices of unseen speakers who are not included in the
training data. Our algorithm alternatively trains two deep neural networks:
multi-task discriminator and multi-speaker neural TTS model (i.e., generator of
GANs). The discriminator is trained not only to distinguish between natural and
synthetic speech but also to verify the speaker of input speech is existent or
non-existent (i.e., newly generated by interpolating seen speakers' embedding
vectors). Meanwhile, the generator is trained to minimize the weighted sum of
the speech reconstruction loss and adversarial loss for fooling the
discriminator, which achieves high-quality multi-speaker TTS even if the target
speaker is unseen. Experimental evaluation shows that our algorithm improves
the quality of synthetic speech better than a conventional GANSpeech algorithm.
