  Weight decay is one of the most widely used forms of regularization in deep
learning, and has been shown to improve generalization and robustness. The
optimization objective driving weight decay is a sum of losses plus a term
proportional to the sum of squared weights. This paper argues that stochastic
gradient descent (SGD) may be an inefficient algorithm for this objective. For
neural networks with ReLU activations, solutions to the weight decay objective
are equivalent to those of a different objective in which the regularization
term is instead a sum of products of $\ell_2$ (not squared) norms of the input
and output weights associated with each ReLU neuron. This alternative (and
effectively equivalent) regularization suggests a novel proximal gradient
algorithm for network training. Theory and experiments support the new training
approach, showing that it can converge much faster to the sparse solutions it
shares with standard weight decay training.
