  In the past few years, large-scale pre-trained vision-language models like
CLIP have achieved tremendous success in various fields. Naturally, how to
transfer the rich knowledge in such huge pre-trained models to downstream tasks
and datasets becomes a hot topic. During downstream adaptation, the most
challenging problems are overfitting and catastrophic forgetting, which can
cause the model to overly focus on the current data and lose more crucial
domain-general knowledge. Existing works use classic regularization techniques
to solve the problems. As solutions become increasingly complex, the
ever-growing storage and inference costs are also a significant problem that
urgently needs to be addressed. While in this paper, we start from an
observation that proper random noise can suppress overfitting and catastrophic
forgetting. Then we regard quantization error as a kind of noise, and explore
quantization for regularizing vision-language model, which is quite efficiency
and effective. Furthermore, to improve the model's generalization capability
while maintaining its specialization capacity at minimal cost, we deeply
analyze the characteristics of the weight distribution in prompts, conclude
several principles for quantization module design and follow such principles to
create several competitive baselines. The proposed method is significantly
efficient due to its inherent lightweight nature, making it possible to adapt
on extremely resource-limited devices. Our method can be fruitfully integrated
into many existing approaches like MaPLe, enhancing accuracy while reducing
storage overhead, making it more powerful yet versatile. Extensive experiments
on 11 datasets shows great superiority of our method sufficiently. Code is
available at https://github.com/beyondhtx/QPrompt.
