  Recent adaptations can boost the low-shot capability of Contrastive
Vision-Language Pre-training (CLIP) by effectively facilitating knowledge
transfer. However, these adaptation methods are usually operated on the global
view of an input image, and thus biased perception of partial local details of
the image. To solve this problem, we propose a Visual Content Refinement (VCR)
before the adaptation calculation during the test stage. Specifically, we first
decompose the test image into different scales to shift the feature extractor's
attention to the details of the image. Then, we select the image view with the
max prediction margin in each scale to filter out the noisy image views, where
the prediction margins are calculated from the pre-trained CLIP model. Finally,
we merge the content of the aforementioned selected image views based on their
scales to construct a new robust representation. Thus, the merged content can
be directly used to help the adapter focus on both global and local parts
without any extra training parameters. We apply our method to 3 popular
low-shot benchmark tasks with 13 datasets and achieve a significant improvement
over state-of-the-art methods. For example, compared to the baseline
(Tip-Adapter) on the few-shot classification task, our method achieves about
2\% average improvement for both training-free and training-need settings.
