  Speech conveys more information than text, as the same word can be uttered in
various voices to convey diverse information. Compared to traditional
text-to-speech (TTS) methods relying on speech prompts (reference speech) for
voice variability, using text prompts (descriptions) is more user-friendly
since speech prompts can be hard to find or may not exist at all. TTS
approaches based on the text prompt face two main challenges: 1) the
one-to-many problem, where not all details about voice variability can be
described in the text prompt, and 2) the limited availability of text prompt
datasets, where vendors and large cost of data labeling are required to write
text prompts for speech. In this work, we introduce PromptTTS 2 to address
these challenges with a variation network to provide variability information of
voice not captured by text prompts, and a prompt generation pipeline to utilize
the large language models (LLM) to compose high quality text prompts.
Specifically, the variation network predicts the representation extracted from
the reference speech (which contains full information about voice variability)
based on the text prompt representation. For the prompt generation pipeline, it
generates text prompts for speech with a speech language understanding model to
recognize voice attributes (e.g., gender, speed) from speech and a large
language model to formulate text prompts based on the recognition results.
Experiments on a large-scale (44K hours) speech dataset demonstrate that
compared to the previous works, PromptTTS 2 generates voices more consistent
with text prompts and supports the sampling of diverse voice variability,
thereby offering users more choices on voice generation. Additionally, the
prompt generation pipeline produces high-quality text prompts, eliminating the
large labeling cost. The demo page of PromptTTS 2 is available online.
