  Contextual embedding-based language models trained on large data sets, such
as BERT and RoBERTa, provide strong performance across a wide range of tasks
and are ubiquitous in modern NLP. It has been observed that fine-tuning these
models on tasks involving data from domains different from that on which they
were pretrained can lead to suboptimal performance. Recent work has explored
approaches to adapt pretrained language models to new domains by incorporating
additional pretraining using domain-specific corpora and task data. We propose
an alternative approach for transferring pretrained language models to new
domains by adapting their tokenizers. We show that domain-specific subword
sequences can be efficiently determined directly from divergences in the
conditional token distributions of the base and domain-specific corpora. In
datasets from four disparate domains, we find adaptive tokenization on a
pretrained RoBERTa model provides >97% of the performance benefits of domain
specific pretraining. Our approach produces smaller models and less training
and inference time than other approaches using tokenizer augmentation. While
adaptive tokenization incurs a 6% increase in model parameters in our
experimentation, due to the introduction of 10k new domain-specific tokens, our
approach, using 64 vCPUs, is 72x faster than further pretraining the language
model on domain-specific corpora on 8 TPUs.
