  Recently, we have seen an increasing interest in the area of speech-to-text
translation. This has led to astonishing improvements in this area. In
contrast, the activities in the area of speech-to-speech translation is still
limited, although it is essential to overcome the language barrier. We believe
that one of the limiting factors is the availability of appropriate training
data. We address this issue by creating LibriS2S, to our knowledge the first
publicly available speech-to-speech training corpus between German and English.
For this corpus, we used independently created audio for German and English
leading to an unbiased pronunciation of the text in both languages. This allows
the creation of a new text-to-speech and speech-to-speech translation model
that directly learns to generate the speech signal based on the pronunciation
of the source language. Using this created corpus, we propose Text-to-Speech
models based on the example of the recently proposed FastSpeech 2 model that
integrates source language information. We do this by adapting the model to
take information such as the pitch, energy or transcript from the source speech
as additional input.
