  Large Language Models (LLMs) serve as repositories of extensive world
knowledge, enabling them to perform tasks such as question-answering and
fact-checking. However, this knowledge can become obsolete as global contexts
change. In this paper, we introduce a novel problem in the realm of continual
learning: Online Continual Knowledge Learning (OCKL). This problem formulation
aims to manage the dynamic nature of world knowledge in LMs under real-time
constraints. We propose a new benchmark and evaluation metric designed to
measure both the rate of new knowledge acquisition and the retention of
previously learned knowledge. Our empirical evaluation, conducted using a
variety of state-of-the-art methods, establishes robust base-lines for OCKL.
Our results reveal that existing continual learning approaches are
unfortunately insufficient for tackling the unique challenges posed by OCKL. We
identify key factors that influence the trade-off between knowledge acquisition
and retention, thereby advancing our understanding of how to train LMs in a
continually evolving environment.
