  Some recent models for Text-to-Speech synthesis aim to transfer the prosody
of a reference utterance to the generated target synthetic speech. This is done
by using a learned embedding of the reference utterance, which is used to
condition speech generation. During training, the reference utterance is
identical to the target utterance. Yet, during synthesis, these models are
often used to transfer prosody from a reference that differs from the text or
speaker being synthesized.
  To address this inconsistency, we propose to use a different, but
prosodically-related, utterance during training too. We believe this should
encourage the model to learn to transfer only those characteristics that the
reference and target have in common. If prosody transfer methods do indeed
transfer prosody they should be able to be trained in the way we propose.
However, results show that a model trained under these conditions performs
significantly worse than one trained using the target utterance as a reference.
To explain this, we hypothesize that prosody transfer models do not learn a
transferable representation of prosody, but rather an utterance-level
representation which is highly dependent on both the reference speaker and
reference text.
