  This paper proposes a zero-shot text-to-speech (TTS) conditioned by a
self-supervised speech-representation model acquired through self-supervised
learning (SSL). Conventional methods with embedding vectors from x-vector or
global style tokens still have a gap in reproducing the speaker characteristics
of unseen speakers. A novel point of the proposed method is the direct use of
the SSL model to obtain embedding vectors from speech representations trained
with a large amount of data. We also introduce the separate conditioning of
acoustic features and a phoneme duration predictor to obtain the disentangled
embeddings between rhythm-based speaker characteristics and
acoustic-feature-based ones. The disentangled embeddings will enable us to
achieve better reproduction performance for unseen speakers and rhythm transfer
conditioned by different speeches. Objective and subjective evaluations showed
that the proposed method can synthesize speech with improved similarity and
achieve speech-rhythm transfer.
