  A wide range of NLP tasks benefit from the fine-tuning of pretrained language
models (PLMs). However, a number of redundant parameters which contribute less
to the downstream task are observed in a directly fine-tuned model. We consider
the gap between pretraining and downstream tasks hinders the training of these
redundant parameters, and results in a suboptimal performance of the overall
model. In this paper, we present PATS (Perturbation According To Sensitivity),
a noisy training mechanism which considers each parameter's importance in the
downstream task to help fine-tune PLMs. The main idea of PATS is to add bigger
noise to parameters with lower sensitivity and vice versa, in order to activate
more parameters' contributions to downstream tasks without affecting the
sensitive ones much. Extensive experiments conducted on different tasks of the
GLUE benchmark show PATS can consistently empower the fine-tuning of different
sizes of PLMs, and the parameters in the well-performing models always have
more concentrated distributions of sensitivities, which experimentally proves
the effectiveness of our method.
