  Pre-trained vision transformers have strong representation benefits to
various downstream tasks. Recently, many parameter-efficient fine-tuning (PEFT)
methods have been proposed, and their experiments demonstrate that tuning only
1\% extra parameters could surpass full fine-tuning in low-data resource
scenarios. However, these methods overlook the task-specific information when
fine-tuning diverse downstream tasks. In this paper, we propose a simple yet
effective method called "Salient Channel Tuning" (SCT) to leverage the
task-specific information by forwarding the model with the task images to
select partial channels in a feature map that enables us to tune only 1/8
channels leading to significantly lower parameter costs. Experiments on 19
visual transfer learning downstream tasks demonstrate that our SCT outperforms
full fine-tuning on 18 out of 19 tasks by adding only 0.11M parameters of the
ViT-B, which is 780$\times$ fewer than its full fine-tuning counterpart.
Furthermore, experiments on domain generalization and few-shot classification
further demonstrate the effectiveness and generic of our approach. The code is
available at https://github.com/showlab/SCT.
