  Zero-shot text-to-speech (TTS) aims to synthesize voices with unseen speech
prompts, which significantly reduces the data and computation requirements for
voice cloning by skipping the fine-tuning process. However, the prompting
mechanisms of zero-shot TTS still face challenges in the following aspects: 1)
previous works of zero-shot TTS are typically trained with single-sentence
prompts, which significantly restricts their performance when the data is
relatively sufficient during the inference stage. 2) The prosodic information
in prompts is highly coupled with timbre, making it untransferable to each
other. This paper introduces Mega-TTS 2, a generic prompting mechanism for
zero-shot TTS, to tackle the aforementioned challenges. Specifically, we design
a powerful acoustic autoencoder that separately encodes the prosody and timbre
information into the compressed latent space while providing high-quality
reconstructions. Then, we propose a multi-reference timbre encoder and a
prosody latent language model (P-LLM) to extract useful information from
multi-sentence prompts. We further leverage the probabilities derived from
multiple P-LLM outputs to produce transferable and controllable prosody.
Experimental results demonstrate that Mega-TTS 2 could not only synthesize
identity-preserving speech with a short prompt of an unseen speaker from
arbitrary sources but consistently outperform the fine-tuning method when the
volume of data ranges from 10 seconds to 5 minutes. Furthermore, our method
enables to transfer various speaking styles to the target timbre in a
fine-grained and controlled manner. Audio samples can be found in
https://boostprompt.github.io/boostprompt/.
