  Vision-language foundation models have exhibited remarkable success across a
multitude of downstream tasks due to their scalability on extensive image-text
paired data. However, these models also display significant limitations when
applied to downstream tasks, such as fine-grained image classification, as a
result of ``decision shortcuts'' that hinder their generalization capabilities.
In this work, we find that the CLIP model possesses a rich set of features,
encompassing both \textit{desired invariant causal features} and
\textit{undesired decision shortcuts}. Moreover, the underperformance of CLIP
on downstream tasks originates from its inability to effectively utilize
pre-trained features in accordance with specific task requirements. To address
this challenge, we propose a simple yet effective method, Spurious Feature
Eraser (SEraser), to alleviate the decision shortcuts by erasing the spurious
features. Specifically, we introduce a test-time prompt tuning paradigm that
optimizes a learnable prompt, thereby compelling the model to exploit invariant
features while disregarding decision shortcuts during the inference phase. The
proposed method effectively alleviates excessive dependence on potentially
misleading spurious information. We conduct comparative analysis of the
proposed method against various approaches which validates the significant
superiority.
