  Adapter-based tuning has recently arisen as an alternative to fine-tuning. It
works by adding light-weight adapter modules to a pretrained language model
(PrLM) and only updating the parameters of adapter modules when learning on a
downstream task. As such, it adds only a few trainable parameters per new task,
allowing a high degree of parameter sharing. Prior studies have shown that
adapter-based tuning often achieves comparable results to fine-tuning. However,
existing work only focuses on the parameter-efficient aspect of adapter-based
tuning while lacking further investigation on its effectiveness. In this paper,
we study the latter. We first show that adapter-based tuning better mitigates
forgetting issues than fine-tuning since it yields representations with less
deviation from those generated by the initial PrLM. We then empirically compare
the two tuning methods on several downstream NLP tasks and settings. We
demonstrate that 1) adapter-based tuning outperforms fine-tuning on
low-resource and cross-lingual tasks; 2) it is more robust to overfitting and
less sensitive to changes in learning rates.
