  Handling distribution shifts from training data, known as out-of-distribution
(OOD) generalization, poses a significant challenge in the field of machine
learning. While a pre-trained vision-language model like CLIP has demonstrated
remarkable zero-shot performance, further adaptation of the model to downstream
tasks leads to undesirable degradation for OOD data. In this work, we introduce
Sparse Adaptation for Fine-Tuning (SAFT), a method that prevents fine-tuning
from forgetting the general knowledge in the pre-trained model. SAFT only
updates a small subset of important parameters whose gradient magnitude is
large, while keeping the other parameters frozen. SAFT is straightforward to
implement and conceptually simple. Extensive experiments show that with only
0.1% of the model parameters, SAFT can significantly improve the performance of
CLIP. It consistently outperforms baseline methods across several benchmarks.
On the few-shot learning benchmark of ImageNet and its variants, SAFT gives a
gain of 5.15% on average over the conventional fine-tuning method in OOD
settings.
