  Existing research has shown that a multilingual pre-trained language model
fine-tuned with one (source) language also performs well on downstream tasks
for non-source languages, even though no fine-tuning is done on these
languages. However, there is a clear gap between the performance of the source
language and that of the non-source languages. This paper analyzes the
fine-tuning process, discovers when the performance gap changes and identifies
which network weights affect the overall performance most. Additionally, the
paper seeks to answer to what extent the gap can be reduced by reducing
forgetting. Based on the analysis results, a method named Fine-tuning slow and
fast with four training policies is proposed to address these issues.
Experimental results show the proposed method outperforms baselines by a clear
margin.
