  This work connects language model adaptation with concepts of machine
learning theory. We consider a training setup with a large out-of-domain set
and a small in-domain set. We derive how the benefit of training a model on
either set depends on the size of the sets and the distance between their
underlying distributions. We analyze how out-of-domain pre-training before
in-domain fine-tuning achieves better generalization than either solution
independently. Finally, we present how adaptation techniques based on data
selection, such as importance sampling, intelligent data selection and
influence functions, can be presented in a common framework which highlights
their similarity and also their subtle differences.
