  Large-scale vision-language models (VLMs) have shown a strong zero-shot
generalization capability on unseen-domain data. However, adapting pre-trained
VLMs to a sequence of downstream tasks often leads to the forgetting of
previously learned knowledge and a reduction in zero-shot classification
performance. To tackle this problem, we propose a unique Selective Dual-Teacher
Knowledge Transfer framework that leverages the most recent fine-tuned and the
original pre-trained VLMs as dual teachers to preserve the previously learned
knowledge and zero-shot capabilities, respectively. With only access to an
unlabeled reference dataset, our proposed framework performs a selective
knowledge distillation mechanism by measuring the feature discrepancy from the
dual-teacher VLMs. Consequently, our selective dual-teacher knowledge
distillation mitigates catastrophic forgetting of previously learned knowledge
while preserving the zero-shot capabilities of pre-trained VLMs. Extensive
experiments on benchmark datasets demonstrate that our framework is favorable
against state-of-the-art continual learning approaches for preventing
catastrophic forgetting and zero-shot degradation. Project page:
https://chuyu.org/research/snd
