  Language models (LMs) have been instrumental for the rapid advance of natural
language processing. This paper studies continual pre-training of LMs, in
particular, continual domain-adaptive pre-training (or continual DAP-training).
Existing research has shown that further pre-training an LM using a domain
corpus to adapt the LM to the domain can improve the end-task performance in
the domain. This paper proposes a novel method to continually DAP-train an LM
with a sequence of unlabeled domain corpora to adapt the LM to these domains to
improve their end-task performances. The key novelty of our method is a
soft-masking mechanism that directly controls the update to the LM. A novel
proxy is also proposed to preserve the general knowledge in the original LM.
Additionally, it contrasts the representations of the previously learned domain
knowledge (including the general knowledge in the pre-trained LM) and the
knowledge from the current full network to achieve knowledge integration. The
method not only overcomes catastrophic forgetting, but also achieves knowledge
transfer to improve end-task performances. Empirical evaluation demonstrates
the effectiveness of the proposed method.
