  Recently, foundation language models (LMs) have marked significant
achievements in the domains of natural language processing (NLP) and computer
vision (CV). Unlike traditional neural network models, foundation LMs obtain a
great ability for transfer learning by acquiring rich commonsense knowledge
through pre-training on extensive unsupervised datasets with a vast number of
parameters. However, they still can not emulate human-like continuous learning
due to catastrophic forgetting. Consequently, various continual learning
(CL)-based methodologies have been developed to refine LMs, enabling them to
adapt to new tasks without forgetting previous knowledge. However, a systematic
taxonomy of existing approaches and a comparison of their performance are still
lacking, which is the gap that our survey aims to fill. We delve into a
comprehensive review, summarization, and classification of the existing
literature on CL-based approaches applied to foundation language models, such
as pre-trained language models (PLMs), large language models (LLMs) and
vision-language models (VLMs). We divide these studies into offline CL and
online CL, which consist of traditional methods, parameter-efficient-based
methods, instruction tuning-based methods and continual pre-training methods.
Offline CL encompasses domain-incremental learning, task-incremental learning,
and class-incremental learning, while online CL is subdivided into hard task
boundary and blurry task boundary settings. Additionally, we outline the
typical datasets and metrics employed in CL research and provide a detailed
analysis of the challenges and future work for LMs-based continual learning.
