  Adaptive text to speech (TTS) can synthesize new voices in zero-shot
scenarios efficiently, by using a well-trained source TTS model without
adapting it on the speech data of new speakers. Considering seen and unseen
speakers have diverse characteristics, zero-shot adaptive TTS requires strong
generalization ability on speaker characteristics, which brings modeling
challenges. In this paper, we develop AdaSpeech 4, a zero-shot adaptive TTS
system for high-quality speech synthesis. We model the speaker characteristics
systematically to improve the generalization on new speakers. Generally, the
modeling of speaker characteristics can be categorized into three steps:
extracting speaker representation, taking this speaker representation as
condition, and synthesizing speech/mel-spectrogram given this speaker
representation. Accordingly, we improve the modeling in three steps: 1) To
extract speaker representation with better generalization, we factorize the
speaker characteristics into basis vectors and extract speaker representation
by weighted combining of these basis vectors through attention. 2) We leverage
conditional layer normalization to integrate the extracted speaker
representation to TTS model. 3) We propose a novel supervision loss based on
the distribution of basis vectors to maintain the corresponding speaker
characteristics in generated mel-spectrograms. Without any fine-tuning,
AdaSpeech 4 achieves better voice quality and similarity than baselines in
multiple datasets.
