  Prompt learning has become a popular approach for adapting large
vision-language models, such as CLIP, to downstream tasks. Typically, prompt
learning relies on a fixed prompt token or an input-conditional token to fit a
small amount of data under full supervision. While this paradigm can generalize
to a certain range of unseen classes, it may struggle when domain gap
increases, such as in fine-grained classification and satellite image
segmentation. To address this limitation, we propose Retrieval-enhanced Prompt
learning (RePrompt), which introduces retrieval mechanisms to cache the
knowledge representations from downstream tasks. we first construct a retrieval
database from training examples, or from external examples when available. We
then integrate this retrieval-enhanced mechanism into various stages of a
simple prompt learning baseline. By referencing similar samples in the training
set, the enhanced model is better able to adapt to new tasks with few samples.
Our extensive experiments over 15 vision datasets, including 11 downstream
tasks with few-shot setting and 4 domain generalization benchmarks, demonstrate
that RePrompt achieves considerably improved performance. Our proposed approach
provides a promising solution to the challenges faced by prompt learning when
domain gap increases. The code and models will be available.
