  With the advent of vision-language models (VLMs) that can perform in-context
and prompt-based learning, how can we design prompting approaches that robustly
generalize to distribution shift and can be used on novel classes outside the
support set of the prompts? In this work, we first define two types of
robustness to distribution shift on VLMs, namely, robustness on base classes
(the classes included in the support set of prompts) and robustness on novel
classes. Then, we study the robustness of existing in-context learning and
prompt learning approaches, where we find that prompt learning performs
robustly on test images from base classes, while it does not generalize well on
images from novel classes. We propose robust prompt learning by integrating
multiple-scale image features into the prompt, which improves both types of
robustness. Comprehensive experiments are conducted to study the defined
robustness on six benchmarks and show the effectiveness of our proposal.
