  Pre-trained language models (LM) such as BERT, DistilBERT, and RoBERTa can be
tuned for different domains (domain-tuning) by continuing the pre-training
phase on a new target domain corpus. This simple domain tuning (SDT) technique
has been widely used to create domain-tuned models such as BioBERT, SciBERT and
ClinicalBERT. However, during the pretraining phase on the target domain, the
LM models may catastrophically forget the patterns learned from their source
domain. In this work, we study the effects of catastrophic forgetting on
domain-tuned LM models and investigate methods that mitigate its negative
effects. We propose continual learning (CL) based alternatives for SDT, that
aim to reduce catastrophic forgetting. We show that these methods may increase
the performance of LM models on downstream target domain tasks. Additionally,
we also show that constraining the LM model from forgetting the source domain
leads to downstream task models that are more robust to domain shifts. We
analyze the computational cost of using our proposed CL methods and provide
recommendations for computationally lightweight and effective CL domain-tuning
procedures.
