  Standard fine-tuning of language models typically performs well on
in-distribution data, but suffers with generalization to distribution shifts.
In this work, we aim to improve the generalization of adapter-based
cross-lingual task transfer where such cross-language distribution shifts are
imminent. We investigate scheduled unfreezing algorithms -- originally proposed
to mitigate catastrophic forgetting in transfer learning -- for fine-tuning
task adapters. Our experiments show that scheduled unfreezing methods close the
gap to full fine-tuning and achieve stronger cross-lingual transfer
performance, suggesting that these methods can go beyond just mitigating
catastrophic forgetting. Next, aiming to understand these empirical findings,
we investigate the learning dynamics of scheduled unfreezing using Fisher
Information. Our experiments reveal that scheduled unfreezing induces different
learning dynamics compared to standard fine-tuning, and provide evidence that
the dynamics of Fisher Information during training correlate with cross-lingual
generalization performance. We additionally propose a general scheduled
unfreezing algorithm that achieves an average of 2 points improvement over four
datasets compared to standard fine-tuning and provides empirical evidence for a
theory-based justification of the heuristic unfreezing schedule for adapter
training.
