  Recent prompt-based text-to-speech (TTS) models can clone an unseen speaker
using only a short speech prompt. They leverage a strong in-context ability to
mimic the speech prompts, including speaker style, prosody, and emotion.
Therefore, the selection of a speech prompt greatly influences the generated
speech, akin to the importance of a prompt in large language models (LLMs).
However, current prompt-based TTS models choose the speech prompt manually or
simply at random. Hence, in this paper, we adapt retrieval augmented generation
(RAG) from LLMs to prompt-based TTS. Unlike traditional RAG methods, we
additionally consider contextual information during the retrieval process and
present a Context-Aware Contrastive Language-Audio Pre-training (CA-CLAP) model
to extract context-aware, style-related features. The objective and subjective
evaluations demonstrate that our proposed RAG method outperforms baselines, and
our CA-CLAP achieves better results than text-only retrieval methods.
