  Zero-shot cross-lingual knowledge transfer enables a multilingual pretrained
language model, finetuned on a task in one language, make predictions for this
task in other languages. While being broadly studied for natural language
understanding tasks, the described setting is understudied for generation.
Previous works notice a frequent problem of generation in a wrong language and
propose approaches to address it, usually using mT5 as a backbone model. In
this work we compare various approaches proposed from the literature in unified
settings, also including alternative backbone models, namely mBART and
NLLB-200. We first underline the importance of tuning learning rate used for
finetuning, which helps to substantially alleviate the problem of generation in
the wrong language. Then, we show that with careful learning rate tuning, the
simple full finetuning of the model acts as a very strong baseline and
alternative approaches bring only marginal improvements. Finally, we find that
mBART performs similarly to mT5 of the same size, and NLLB-200 can be
competitive in some cases. Our final zero-shot models reach the performance of
the approach based on data translation which is usually considered as an upper
baseline for zero-shot cross-lingual transfer in generation.
