  Training language models becomes increasingly expensive with scale, prompting
numerous attempts to improve optimization efficiency. Despite these efforts,
the Adam optimizer remains the most widely used, due to a prevailing view that
it is the most effective approach. We aim to compare several optimization
algorithms, including SGD, Adafactor, Adam, and Lion, in the context of
autoregressive language modeling across a range of model sizes,
hyperparameters, and architecture variants. Our findings indicate that, except
for SGD, these algorithms all perform comparably both in their optimal
performance and also in terms of how they fare across a wide range of
hyperparameter choices. Our results suggest to practitioners that the choice of
optimizer can be guided by practical considerations like memory constraints and
ease of implementation, as no single algorithm emerged as a clear winner in
terms of performance or stability to hyperparameter misspecification.
  Given our findings, we further dissect these approaches, examining two
simplified versions of Adam: a) signed momentum (Signum) which we see recovers
both the performance and hyperparameter stability of Adam and b) Adalayer, a
layerwise variant of Adam which we introduce to study Adam's preconditioning.
Examining Adalayer leads us to the conclusion that the largest impact of Adam's
preconditioning is restricted to the last layer and LayerNorm parameters, and,
perhaps surprisingly, the remaining layers can be trained with SGD.
