  It has been known that direct speech-to-speech translation (S2ST) models
usually suffer from the data scarcity issue because of the limited existing
parallel materials for both source and target speech. Therefore to train a
direct S2ST system, previous works usually utilize text-to-speech (TTS) systems
to generate samples in the target language by augmenting the data from
speech-to-text translation (S2TT). However, there is a limited investigation
into how the synthesized target speech would affect the S2ST models. In this
work, we analyze the effect of changing synthesized target speech for direct
S2ST models. We find that simply combining the target speech from different TTS
systems can potentially improve the S2ST performances. Following that, we also
propose a multi-task framework that jointly optimizes the S2ST system with
multiple targets from different TTS systems. Extensive experiments demonstrate
that our proposed framework achieves consistent improvements (2.8 BLEU) over
the baselines on the Fisher Spanish-English dataset.
