  Weight decay is one of the standard tricks in the neural network toolbox, but
the reasons for its regularization effect are poorly understood, and recent
results have cast doubt on the traditional interpretation in terms of $L_2$
regularization. Literal weight decay has been shown to outperform $L_2$
regularization for optimizers for which they differ. We empirically investigate
weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a
variety of network architectures. We identify three distinct mechanisms by
which weight decay exerts a regularization effect, depending on the particular
optimization algorithm and architecture: (1) increasing the effective learning
rate, (2) approximately regularizing the input-output Jacobian norm, and (3)
reducing the effective damping coefficient for second-order optimization. Our
results provide insight into how to improve the regularization of neural
networks.
