  The task of Few-shot Learning (FSL) aims to do the inference on novel
categories containing only few labeled examples, with the help of knowledge
learned from base categories containing abundant labeled training samples.
While there are numerous works into FSL task, Vision Transformers (ViTs) have
rarely been taken as the backbone to FSL with few trials focusing on naive
finetuning of whole backbone or classification layer.} Essentially, despite
ViTs have been shown to enjoy comparable or even better performance on other
vision tasks, it is still very nontrivial to efficiently finetune the ViTs in
real-world FSL scenarios. To this end, we propose a novel efficient Transformer
Tuning (eTT) method that facilitates finetuning ViTs in the FSL tasks. The key
novelties come from the newly presented Attentive Prefix Tuning (APT) and
Domain Residual Adapter (DRA) for the task and backbone tuning, individually.
Specifically, in APT, the prefix is projected to new key and value pairs that
are attached to each self-attention layer to provide the model with
task-specific information. Moreover, we design the DRA in the form of learnable
offset vectors to handle the potential domain gaps between base and novel data.
To ensure the APT would not deviate from the initial task-specific information
much, we further propose a novel prototypical regularization, which maximizes
the similarity between the projected distribution of prefix and initial
prototypes, regularizing the update procedure. Our method receives outstanding
performance on the challenging Meta-Dataset. We conduct extensive experiments
to show the efficacy of our model.
