  The performance of fine-tuning pre-trained language models largely depends on
the hyperparameter configuration. In this paper, we investigate the performance
of modern hyperparameter optimization methods (HPO) on fine-tuning pre-trained
language models. First, we study and report three HPO algorithms' performances
on fine-tuning two state-of-the-art language models on the GLUE dataset. We
find that using the same time budget, HPO often fails to outperform grid search
due to two reasons: insufficient time budget and overfitting. We propose two
general strategies and an experimental procedure to systematically troubleshoot
HPO's failure cases. By applying the procedure, we observe that HPO can succeed
with more appropriate settings in the search space and time budget; however, in
certain cases overfitting remains. Finally, we make suggestions for future
work. Our implementation can be found in
https://github.com/microsoft/FLAML/tree/main/flaml/nlp/.
