  Contrastive vision-language models like CLIP have shown great progress in
transfer learning. In the inference stage, the proper text description, also
known as prompt, needs to be carefully designed to correctly classify the given
images. In order to avoid laborious prompt engineering, recent works such as
CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for
downstream image recognition tasks on a small set of labeled data. Though
promising improvements are achieved, requiring labeled data from the target
datasets may restrict the scalability. In this paper, we explore a different
scenario, in which the labels of the target datasets are unprovided, and we
present an unsupervised prompt learning (UPL) approach to avoid prompt
engineering while simultaneously improving transfer performance of CLIP-like
vision-language models. As far as we know, UPL is the first work to introduce
unsupervised learning into prompt learning. Experimentally, our UPL outperforms
original CLIP with prompt engineering on ImageNet as well as other 10 datasets.
An enhanced version of UPL is even competitive with the 8-shot CoOp and the
8-shot TIP-Adapter on most datasets. Code and models are available at
https://github.com/tonyhuang2022/UPL.
