  Large Language Models (LMs) are known to encode world knowledge in their
parameters as they pretrain on a vast amount of web corpus, which is often
utilized for performing knowledge-dependent downstream tasks such as question
answering, fact-checking, and open dialogue. In real-world scenarios, the world
knowledge stored in the LMs can quickly become outdated as the world changes,
but it is non-trivial to avoid catastrophic forgetting and reliably acquire new
knowledge while preserving invariant knowledge. To push the community towards
better maintenance of ever-changing LMs, we formulate a new continual learning
(CL) problem called Continual Knowledge Learning (CKL). We construct a new
benchmark and metric to quantify the retention of time-invariant world
knowledge, the update of outdated knowledge, and the acquisition of new
knowledge. We adopt applicable recent methods from literature to create several
strong baselines. Through extensive experiments, we find that CKL exhibits
unique challenges that are not addressed in previous CL setups, where parameter
expansion is necessary to reliably retain and learn knowledge simultaneously.
By highlighting the critical causes of knowledge forgetting, we show that CKL
is a challenging and important problem that helps us better understand and
train ever-changing LMs. The benchmark datasets, evaluation script, and
baseline code to reproduce our results are available at
https://github.com/joeljang/continual-knowledge-learning.
