  Prompt tuning, which involves training a small set of parameters, effectively
enhances the pre-trained Vision-Language Models (VLMs) to downstream tasks.
However, they often come at the cost of flexibility and adaptability when the
tuned models are applied to different datasets or domains. In this paper, we
explore capturing the task-specific information via meticulous refinement of
entire VLMs, with minimal parameter adjustments. When fine-tuning the entire
VLMs for specific tasks under limited supervision, overfitting and catastrophic
forgetting become the defacto factors. To mitigate these issues, we propose a
framework named CLIP-CITE via designing a discriminative visual-text task,
further aligning the visual-text semantics in a supervision manner, and
integrating knowledge distillation techniques to preserve the gained knowledge.
Extensive experimental results under few-shot learning, base-to-new
generalization, domain generalization, and cross-domain generalization
settings, demonstrate that our method effectively enhances the performance on
specific tasks under limited supervision while preserving the versatility of
the VLMs on other datasets.
