  Fine-tuning vision-language models (VLMs) with abundant unlabeled data
recently has attracted increasing attention. Existing methods that resort to
the pseudolabeling strategy would suffer from heavily incorrect hard
pseudolabels when VLMs exhibit low zero-shot performance in downstream tasks.
To alleviate this issue, we propose a Candidate Pseudolabel Learning method,
termed CPL, to fine-tune VLMs with suitable candidate pseudolabels of unlabeled
data in downstream tasks. The core of our method lies in the generation
strategy of candidate pseudolabels, which progressively generates refined
candidate pseudolabels by both intra- and inter-instance label selection, based
on a confidence score matrix for all unlabeled data. This strategy can result
in better performance in true label inclusion and class-balanced instance
selection. In this way, we can directly apply existing loss functions to learn
with generated candidate psueudolabels. Extensive experiments on nine benchmark
datasets with three learning paradigms demonstrate the effectiveness of our
method. Our code can be found at https://github.com/vanillaer/CPL-ICML2024.
