  Fine-tuning large pretrained language models on a limited training corpus
usually suffers from poor generalization. Prior works show that the
recently-proposed sharpness-aware minimization (SAM) optimization method can
improve the model generalization. However, SAM adds a perturbation to each
model parameter equally (but not all parameters contribute equally to the
optimization of training), which we argue is sub-optimal and will lead to
excessive computation. In this paper, we propose a novel optimization
procedure, namely FSAM, which introduces a Fisher mask to improve the
efficiency and performance of SAM. In short, instead of adding perturbation to
all parameters, FSAM uses the Fisher information to identity the important
parameters and formulates a Fisher mask to obtain the sparse perturbation,
i.e., making the optimizer focus on these important parameters. Experiments on
various tasks in GLUE and SuperGLUE benchmarks show that FSAM consistently
outperforms the vanilla SAM by 0.67~1.98 average score among four different
pretrained models. We also empirically show that FSAM works well in other
complex scenarios, e.g., fine-tuning on generation tasks or limited training
data. Encouragingly, when training data is limited, FSAM improves the SAM by a
large margin, i.e., up to 15.1.
