  In this work, we induce character-level noise in various forms when
fine-tuning BERT to enable zero-shot cross-lingual transfer to unseen dialects
and languages. We fine-tune BERT on three sentence-level classification tasks
and evaluate our approach on an assortment of unseen dialects and languages. We
find that character-level noise can be an extremely effective agent of
cross-lingual transfer under certain conditions, while it is not as helpful in
others. Specifically, we explore these differences in terms of the nature of
the task and the relationships between source and target languages, finding
that introduction of character-level noise during fine-tuning is particularly
helpful when a task draws on surface level cues and the source-target
cross-lingual pair has a relatively high lexical overlap with shorter (i.e.,
less meaningful) unseen tokens on average.
