  NLP research has explored different neural model architectures and sizes,
datasets, training objectives, and transfer learning techniques. However, the
choice of optimizer during training has not been explored as extensively.
Typically, some variant of Stochastic Gradient Descent (SGD) is employed,
selected among numerous variants, using unclear criteria, often with minimal or
no tuning of the optimizer's hyperparameters. Experimenting with five GLUE
datasets, two models (DistilBERT and DistilRoBERTa), and seven popular
optimizers (SGD, SGD with Momentum, Adam, AdaMax, Nadam, AdamW, and AdaBound),
we find that when the hyperparameters of the optimizers are tuned, there is no
substantial difference in test performance across the five more elaborate
(adaptive) optimizers, despite differences in training loss. Furthermore,
tuning just the learning rate is in most cases as good as tuning all the
hyperparameters. Hence, we recommend picking any of the best-behaved adaptive
optimizers (e.g., Adam) and tuning only its learning rate. When no
hyperparameter can be tuned, SGD with Momentum is the best choice.
