  Cross-lingual Summarization (CLS) aims at producing a summary in the target
language for an article in the source language. Traditional solutions employ a
two-step approach, i.e. translate then summarize or summarize then translate.
Recently, end-to-end models have achieved better results, but these approaches
are mostly limited by their dependence on large-scale labeled data. We propose
a solution based on mixed-lingual pre-training that leverages both
cross-lingual tasks such as translation and monolingual tasks like masked
language models. Thus, our model can leverage the massive monolingual data to
enhance its modeling of language. Moreover, the architecture has no
task-specific components, which saves memory and increases optimization
efficiency. We show in experiments that this pre-training scheme can
effectively boost the performance of cross-lingual summarization. In Neural
Cross-Lingual Summarization (NCLS) dataset, our model achieves an improvement
of 2.82 (English to Chinese) and 1.15 (Chinese to English) ROUGE-1 scores over
state-of-the-art results.
