  While recent text-to-speech (TTS) systems have made remarkable strides toward
human-level quality, the performance of cross-lingual TTS lags behind that of
intra-lingual TTS. This gap is mainly rooted from the speaker-language
entanglement problem in cross-lingual TTS. In this paper, we propose
CrossSpeech which improves the quality of cross-lingual speech by effectively
disentangling speaker and language information in the level of acoustic feature
space. Specifically, CrossSpeech decomposes the speech generation pipeline into
the speaker-independent generator (SIG) and speaker-dependent generator (SDG).
The SIG produces the speaker-independent acoustic representation which is not
biased to specific speaker distributions. On the other hand, the SDG models
speaker-dependent speech variation that characterizes speaker attributes. By
handling each information separately, CrossSpeech can obtain disentangled
speaker and language representations. From the experiments, we verify that
CrossSpeech achieves significant improvements in cross-lingual TTS, especially
in terms of speaker similarity to the target speaker.
