  In this paper, we explore the challenging problem of performing a generative
task in a target language when labeled data is only available in English, using
summarization as a case study. We assume a strict setting with no access to
parallel data or machine translation and find that common transfer learning
approaches struggle in this setting, as a generative multilingual model
fine-tuned purely on English catastrophically forgets how to generate
non-English. Given the recent rise of parameter-efficient adaptation
techniques, we conduct the first investigation into how one such method, prompt
tuning (Lester et al., 2021), can overcome catastrophic forgetting to enable
zero-shot cross-lingual generation. Our experiments show that
parameter-efficient prompt tuning provides gains over standard fine-tuning when
transferring between less-related languages, e.g., from English to Thai.
However, a significant gap still remains between these methods and
fully-supervised baselines. To improve cross-lingual transfer further, we
explore several approaches, including: (1) mixing in unlabeled multilingual
data, and (2) explicitly factoring prompts into recombinable language and task
components. Our approaches can provide further quality gains, suggesting that
robust zero-shot cross-lingual generation is within reach.
