  Continual learning (CL) can help pre-trained vision-language models
efficiently adapt to new or under-trained data distributions without
re-training. Nevertheless, during the continual training of the Contrastive
Language-Image Pre-training (CLIP) model, we observe that the model's zero-shot
transfer ability significantly degrades due to catastrophic forgetting.
Existing CL methods can mitigate forgetting by replaying previous data.
However, since the CLIP dataset is private, replay methods cannot access the
pre-training dataset. In addition, replaying data of previously learned
downstream tasks can enhance their performance but comes at the cost of
sacrificing zero-shot performance. To address this challenge, we propose a
novel method ZSCL to prevent zero-shot transfer degradation in the continual
learning of vision-language models in both feature and parameter space. In the
feature space, a reference dataset is introduced for distillation between the
current and initial models. The reference dataset should have semantic
diversity but no need to be labeled, seen in pre-training, or matched
image-text pairs. In parameter space, we prevent a large parameter shift by
averaging weights during the training. We propose a more challenging
Multi-domain Task Incremental Learning (MTIL) benchmark to evaluate different
methods, where tasks are from various domains instead of class-separated in a
single dataset. Our method outperforms other methods in the traditional
class-incremental learning setting and the MTIL by 9.7% average score. Our code
locates at https://github.com/Thunderbeee/ZSCL.
