  Large-scale pre-trained language models have been shown to be helpful in
improving the naturalness of text-to-speech (TTS) models by enabling them to
produce more naturalistic prosodic patterns. However, these models are usually
word-level or sup-phoneme-level and jointly trained with phonemes, making them
inefficient for the downstream TTS task where only phonemes are needed. In this
work, we propose a phoneme-level BERT (PL-BERT) with a pretext task of
predicting the corresponding graphemes along with the regular masked phoneme
predictions. Subjective evaluations show that our phoneme-level BERT encoder
has significantly improved the mean opinion scores (MOS) of rated naturalness
of synthesized speech compared with the state-of-the-art (SOTA) StyleTTS
baseline on out-of-distribution (OOD) texts.
