  Pre-trained vision-language models have inspired much research on few-shot
learning. However, with only a few training images, there exist two crucial
problems: (1) the visual feature distributions are easily distracted by
class-irrelevant information in images, and (2) the alignment between the
visual and language feature distributions is difficult. To deal with the
distraction problem, we propose a Selective Attack module, which consists of
trainable adapters that generate spatial attention maps of images to guide the
attacks on class-irrelevant image areas. By messing up these areas, the
critical features are captured and the visual distributions of image features
are calibrated. To better align the visual and language feature distributions
that describe the same object class, we propose a cross-modal distribution
alignment module, in which we introduce a vision-language prototype for each
class to align the distributions, and adopt the Earth Mover's Distance (EMD) to
optimize the prototypes. For efficient computation, the upper bound of EMD is
derived. In addition, we propose an augmentation strategy to increase the
diversity of the images and the text prompts, which can reduce overfitting to
the few-shot training images. Extensive experiments on 11 datasets demonstrate
that our method consistently outperforms prior arts in few-shot learning. The
implementation code will be available at https://github.com/bhrqw/SADA.
