  Vision-language models (VLMs) have emerged as formidable tools, showing their
strong capability in handling various open-vocabulary tasks in image
recognition, text-driven visual content generation, and visual chatbots, to
name a few. In recent years, considerable efforts and resources have been
devoted to adaptation methods for improving downstream performance of VLMs,
particularly on parameter-efficient fine-tuning methods like prompt learning.
However, a crucial aspect that has been largely overlooked is the confidence
calibration problem in fine-tuned VLMs, which could greatly reduce reliability
when deploying such models in the real world. This paper bridges the gap by
systematically investigating the confidence calibration problem in the context
of prompt learning and reveals that existing calibration methods are
insufficient to address the problem, especially in the open-vocabulary setting.
To solve the problem, we present a simple and effective approach called
Distance-Aware Calibration (DAC), which is based on scaling the temperature
using as guidance the distance between predicted text labels and base classes.
The experiments with 7 distinct prompt learning methods applied across 11
diverse downstream datasets demonstrate the effectiveness of DAC, which
achieves high efficacy without sacrificing the inference speed. Our code is
available at https://github.com/ml-stat-Sustech/CLIP_Calibration.
