  We present a suite of experiments that allow us to understand the underlying
challenges of language model adaptation to nonstandard text. We do so by
designing interventions that approximate several types of linguistic variation
and their interactions with existing biases of language models. Applying our
interventions during language model adaptation with varying size and nature of
training data, we gain important insights into when knowledge transfer can be
successful, as well as the aspects of linguistic variation that are
particularly difficult for language models to deal with. For instance, on text
with character-level variation, performance improves with even a few training
examples but approaches a plateau, suggesting that more data is not the
solution. In contrast, on text with variation involving new words or meanings,
far more data is needed, but it leads to a massive breakthrough in performance.
Our findings reveal that existing models lack the necessary infrastructure to
handle diverse forms of nonstandard text and linguistic variation, guiding the
development of more resilient language modeling techniques for the future. We
make the code for our interventions, which can be applied to any English text
data, publicly available.
