  Although soft prompt tuning is effective in efficiently adapting
Vision-Language (V&L) models for downstream tasks, it shows limitations in
dealing with distribution shifts. We address this issue with Attribute-Guided
Prompt Tuning (ArGue), making three key contributions. 1) In contrast to the
conventional approach of directly appending soft prompts preceding class names,
we align the model with primitive visual attributes generated by Large Language
Models (LLMs). We posit that a model's ability to express high confidence in
these attributes signifies its capacity to discern the correct class
rationales. 2) We introduce attribute sampling to eliminate disadvantageous
attributes, thus only semantically meaningful attributes are preserved. 3) We
propose negative prompting, explicitly enumerating class-agnostic attributes to
activate spurious correlations and encourage the model to generate highly
orthogonal probability distributions in relation to these negative features. In
experiments, our method significantly outperforms current state-of-the-art
prompt tuning methods on both novel class prediction and out-of-distribution
generalization tasks.
