  Vision-language models (VLMs) can learn high-quality representations from a
large-scale training dataset of image-text pairs. Prompt learning is a popular
approach to fine-tuning VLM to adapt them to downstream tasks. Despite the
satisfying performance, a major limitation of prompt learning is the demand for
labelled data. In real-world scenarios, we may only obtain candidate labels
(where the true label is included) instead of the true labels due to data
privacy or sensitivity issues. In this paper, we provide the first study on
prompt learning with candidate labels for VLMs. We empirically demonstrate that
prompt learning is more advantageous than other fine-tuning methods, for
handling candidate labels. Nonetheless, its performance drops when the label
ambiguity increases. In order to improve its robustness, we propose a simple
yet effective framework that better leverages the prior knowledge of VLMs to
guide the learning process with candidate labels. Specifically, our framework
disambiguates candidate labels by aligning the model output with the mixed
class posterior jointly predicted by both the learnable and the handcrafted
prompt. Besides, our framework can be equipped with various off-the-shelf
training objectives for learning with candidate labels to further improve their
performance. Extensive experiments demonstrate the effectiveness of our
proposed framework.
