  Through prompting, large-scale pre-trained models have become more expressive
and powerful, gaining significant attention in recent years. Though these big
models have zero-shot capabilities, in general, labeled data are still required
to adapt them to downstream tasks. To overcome this critical limitation, we
propose an unsupervised fine-tuning framework to directly fine-tune the model
or prompt on the unlabeled target data. We demonstrate how to apply our method
to both language-augmented vision and masked-language models by aligning the
discrete distributions extracted from the prompts and target data. To verify
our approach's applicability, we conduct extensive experiments on image
classification, sentiment analysis, and natural language inference tasks.
Across 13 image-related tasks and 15 language-related ones, the proposed
approach achieves consistent improvements over the baselines.
