  Weight decay is a broadly used technique for training state-of-the-art deep
networks, including large language models. Despite its widespread usage, its
role remains poorly understood. In this work, we highlight that the role of
weight decay in modern deep learning is different from its regularization
effect studied in classical learning theory. For overparameterized deep
networks, we show how weight decay modifies the optimization dynamics enhancing
the ever-present implicit regularization of SGD via the loss stabilization
mechanism. In contrast, for underparameterized large language models trained
with nearly online SGD, we describe how weight decay balances the bias-variance
tradeoff in stochastic optimization leading to lower training loss. Moreover,
we show that weight decay also prevents sudden loss divergences for bfloat16
mixed-precision training which is a crucial tool for LLM training. Overall, we
present a unifying perspective from ResNets on vision tasks to LLMs: weight
decay is never useful as an explicit regularizer but instead changes the
training dynamics in a desirable way. Our code is available at
https://github.com/tml-epfl/why-weight-decay.
