  The pretrain-then-finetune paradigm has been widely adopted in computer
vision. But as the size of Vision Transformer (ViT) grows exponentially, the
full finetuning becomes prohibitive in view of the heavier storage overhead.
Motivated by parameter-efficient transfer learning (PETL) on language
transformers, recent studies attempt to insert lightweight adaptation modules
(e.g., adapter layers or prompt tokens) to pretrained ViT and only finetune
these modules while the pretrained weights are frozen. However, these modules
were originally proposed to finetune language models and did not take into
account the prior knowledge specifically for visual tasks. In this paper, we
propose to construct Convolutional Bypasses (Convpass) in ViT as adaptation
modules, introducing only a small amount (less than 0.5% of model parameters)
of trainable parameters to adapt the large ViT. Different from other PETL
methods, Convpass benefits from the hard-coded inductive bias of convolutional
layers and thus is more suitable for visual tasks, especially in the low-data
regime. Experimental results on VTAB-1K benchmark and few-shot learning
datasets show that Convpass outperforms current language-oriented adaptation
modules, demonstrating the necessity to tailor vision-oriented adaptation
modules for adapting vision models.
