  This paper proposes an audio-conditioned phonemic and prosodic annotation
model for building text-to-speech (TTS) datasets from unlabeled speech samples.
For creating a TTS dataset that consists of label-speech paired data, the
proposed annotation model leverages an automatic speech recognition (ASR) model
to obtain phonemic and prosodic labels from unlabeled speech samples. By
fine-tuning a large-scale pre-trained ASR model, we can construct the
annotation model using a limited amount of label-speech paired data within an
existing TTS dataset. To alleviate the shortage of label-speech paired data for
training the annotation model, we generate pseudo label-speech paired data
using text-only corpora and an auxiliary TTS model. This TTS model is also
trained with the existing TTS dataset. Experimental results show that the TTS
model trained with the dataset created by the proposed annotation method can
synthesize speech as naturally as the one trained with a fully-labeled dataset.
