  Multi-speaker speech synthesis is a technique for modeling multiple speakers'
voices with a single model. Although many approaches using deep neural networks
(DNNs) have been proposed, DNNs are prone to overfitting when the amount of
training data is limited. We propose a framework for multi-speaker speech
synthesis using deep Gaussian processes (DGPs); a DGP is a deep architecture of
Bayesian kernel regressions and thus robust to overfitting. In this framework,
speaker information is fed to duration/acoustic models using speaker codes. We
also examine the use of deep Gaussian process latent variable models (DGPLVMs).
In this approach, the representation of each speaker is learned simultaneously
with other model parameters, and therefore the similarity or dissimilarity of
speakers is considered efficiently. We experimentally evaluated two situations
to investigate the effectiveness of the proposed methods. In one situation, the
amount of data from each speaker is balanced (speaker-balanced), and in the
other, the data from certain speakers are limited (speaker-imbalanced).
Subjective and objective evaluation results showed that both the DGP and DGPLVM
synthesize multi-speaker speech more effective than a DNN in the
speaker-balanced situation. We also found that the DGPLVM outperforms the DGP
significantly in the speaker-imbalanced situation.
