  This paper addresses the critical issue of miscalibration in CLIP-based model
adaptation, particularly in the challenging scenario of out-of-distribution
(OOD) samples, which has been overlooked in the existing literature on CLIP
adaptation. We empirically demonstrate that popular CLIP adaptation approaches,
such as Adapters, Prompt Learning, and Test-Time Adaptation, substantially
degrade the calibration capabilities of the zero-shot baseline in the presence
of distributional drift. We identify the increase in logit ranges as the
underlying cause of miscalibration of CLIP adaptation methods, contrasting with
previous work on calibrating fully-supervised models. Motivated by these
observations, we present a simple and model-agnostic solution to mitigate
miscalibration, by scaling the logit range of each sample to its zero-shot
prediction logits. We explore three different alternatives to achieve this,
which can be either integrated during adaptation or directly used at inference
time. Comprehensive experiments on popular OOD classification benchmarks
demonstrate the effectiveness of the proposed approaches in mitigating
miscalibration while maintaining discriminative performance, whose improvements
are consistent across the three families of these increasingly popular
approaches. The code is publicly available at:
https://github.com/Bala93/CLIPCalib
