  Pretrained language models (PTLMs) are typically learned over a large, static
corpus and further fine-tuned for various downstream tasks. However, when
deployed in the real world, a PTLM-based model must deal with data
distributions that deviate from what the PTLM was initially trained on. In this
paper, we study a lifelong language model pretraining challenge where a PTLM is
continually updated so as to adapt to emerging data. Over a domain-incremental
research paper stream and a chronologically-ordered tweet stream, we
incrementally pretrain a PTLM with different continual learning algorithms, and
keep track of the downstream task performance (after fine-tuning). We evaluate
PTLM's ability to adapt to new corpora while retaining learned knowledge in
earlier corpora. Our experiments show distillation-based approaches to be most
effective in retaining downstream performance in earlier domains. The
algorithms also improve knowledge transfer, allowing models to achieve better
downstream performance over the latest data, and improve temporal
generalization when distribution gaps exist between training and evaluation
because of time. We believe our problem formulation, methods, and analysis will
inspire future studies towards continual pretraining of language models.
