  While most research into speech synthesis has focused on synthesizing
high-quality speech for in-dataset speakers, an equally essential yet unsolved
problem is synthesizing speech for unseen speakers who are out-of-dataset with
limited reference data, i.e., speaker adaptive speech synthesis. Many studies
have proposed zero-shot speaker adaptive text-to-speech and voice conversion
approaches aimed at this task. However, most current approaches suffer from the
degradation of naturalness and speaker similarity when synthesizing speech for
unseen speakers (i.e., speakers not in the training dataset) due to the poor
generalizability of the model in out-of-distribution data. To address this
problem, we propose GZS-TV, a generalizable zero-shot speaker adaptive
text-to-speech and voice conversion model. GZS-TV introduces disentangled
representation learning for both speaker embedding extraction and timbre
transformation to improve model generalization and leverages the representation
learning capability of the variational autoencoder to enhance the speaker
encoder. Our experiments demonstrate that GZS-TV reduces performance
degradation on unseen speakers and outperforms all baseline models in multiple
datasets.
