  Parameter-Efficient Tuning (PETuning) methods have been deemed by many as the
new paradigm for using pretrained language models (PLMs). By tuning just a
fraction amount of parameters comparing to full model finetuning, PETuning
methods claim to have achieved performance on par with or even better than
finetuning. In this work, we take a step back and re-examine these PETuning
methods by conducting the first comprehensive investigation into the training
and evaluation of them. We found the problematic validation and testing
practice in current studies, when accompanied by the instability nature of
PETuning methods, has led to unreliable conclusions. When being compared under
a truly fair evaluation protocol, PETuning cannot yield consistently
competitive performance while finetuning remains to be the best-performing
method in medium- and high-resource settings. We delve deeper into the cause of
the instability and observed that the number of trainable parameters and
training iterations are two main factors: reducing trainable parameters and
prolonging training iterations may lead to higher stability in PETuning
methods.
