  Speech representation learning has improved both speech understanding and
speech synthesis tasks for single language. However, its ability in
cross-lingual scenarios has not been explored. In this paper, we extend the
pretraining method for cross-lingual multi-speaker speech synthesis tasks,
including cross-lingual multi-speaker voice cloning and cross-lingual
multi-speaker speech editing. We propose a speech-text joint pretraining
framework, where we randomly mask the spectrogram and the phonemes given a
speech example and its transcription. By learning to reconstruct the masked
parts of the input in different languages, our model shows great improvements
over speaker-embedding-based multi-speaker TTS methods. Moreover, our framework
is end-to-end for both the training and the inference without any finetuning
effort. In cross-lingual multi-speaker voice cloning and cross-lingual
multi-speaker speech editing tasks, our experiments show that our model
outperforms speaker-embedding-based multi-speaker TTS methods.
