  Transferring a pretrained model to a downstream task can be as easy as
conducting linear probing with target data, that is, training a linear
classifier upon frozen features extracted from the pretrained model. As there
may exist significant gaps between pretraining and downstream datasets, one may
ask whether all dimensions of the pretrained features are useful for a given
downstream task. We show that, for linear probing, the pretrained features can
be extremely redundant when the downstream data is scarce, or few-shot. For
some cases such as 5-way 1-shot tasks, using only 1\% of the most important
feature dimensions is able to recover the performance achieved by using the
full representation. Interestingly, most dimensions are redundant only under
few-shot settings and gradually become useful when the number of shots
increases, suggesting that feature redundancy may be the key to characterizing
the "few-shot" nature of few-shot transfer problems. We give a theoretical
understanding of this phenomenon and show how dimensions with high variance and
small distance between class centroids can serve as confounding factors that
severely disturb classification results under few-shot settings. As an attempt
at solving this problem, we find that the redundant features are difficult to
identify accurately with a small number of training samples, but we can instead
adjust feature magnitude with a soft mask based on estimated feature
importance. We show that this method can generally improve few-shot transfer
performance across various pretrained models and downstream datasets.
