  Prompt tuning and adapter tuning have shown great potential in transferring
pre-trained vision-language models (VLMs) to various downstream tasks. In this
work, we design a new type of tuning method, termed as regularized mask tuning,
which masks the network parameters through a learnable selection. Inspired by
neural pathways, we argue that the knowledge required by a downstream task
already exists in the pre-trained weights but just gets concealed in the
upstream pre-training stage. To bring the useful knowledge back into light, we
first identify a set of parameters that are important to a given downstream
task, then attach a binary mask to each parameter, and finally optimize these
masks on the downstream data with the parameters frozen. When updating the
mask, we introduce a novel gradient dropout strategy to regularize the
parameter selection, in order to prevent the model from forgetting old
knowledge and overfitting the downstream data. Experimental results on 11
datasets demonstrate the consistent superiority of our method over previous
alternatives. It is noteworthy that we manage to deliver 18.73% performance
improvement compared to the zero-shot CLIP via masking an average of only 2.56%
parameters. Furthermore, our method is synergistic with most existing
parameter-efficient tuning methods and can boost the performance on top of
them. Project page can be found here (https://wuw2019.github.io/R-AMT/).
