  Normalization techniques are a boon for modern deep learning. They let
weights converge more quickly with often better generalization performances. It
has been argued that the normalization-induced scale invariance among the
weights provides an advantageous ground for gradient descent (GD) optimizers:
the effective step sizes are automatically reduced over time, stabilizing the
overall training procedure. It is often overlooked, however, that the
additional introduction of momentum in GD optimizers results in a far more
rapid reduction in effective step sizes for scale-invariant weights, a
phenomenon that has not yet been studied and may have caused unwanted side
effects in the current practice. This is a crucial issue because arguably the
vast majority of modern deep neural networks consist of (1) momentum-based GD
(e.g. SGD or Adam) and (2) scale-invariant parameters. In this paper, we verify
that the widely-adopted combination of the two ingredients lead to the
premature decay of effective step sizes and sub-optimal model performances. We
propose a simple and effective remedy, SGDP and AdamP: get rid of the radial
component, or the norm-increasing direction, at each optimizer step. Because of
the scale invariance, this modification only alters the effective step sizes
without changing the effective update directions, thus enjoying the original
convergence properties of GD optimizers. Given the ubiquity of momentum GD and
scale invariance in machine learning, we have evaluated our methods against the
baselines on 13 benchmarks. They range from vision tasks like classification
(e.g. ImageNet), retrieval (e.g. CUB and SOP), and detection (e.g. COCO) to
language modelling (e.g. WikiText) and audio classification (e.g. DCASE) tasks.
We verify that our solution brings about uniform gains in those benchmarks.
Source code is available at https://github.com/clovaai/AdamP.
