  Momentum is known to accelerate the convergence of gradient descent in
strongly convex settings without stochastic gradient noise. In stochastic
optimization, such as training neural networks, folklore suggests that momentum
may help deep learning optimization by reducing the variance of the stochastic
gradient update, but previous theoretical analyses do not find momentum to
offer any provable acceleration. Theoretical results in this paper clarify the
role of momentum in stochastic settings where the learning rate is small and
gradient noise is the dominant source of instability, suggesting that SGD with
and without momentum behave similarly in the short and long time horizons.
Experiments show that momentum indeed has limited benefits for both
optimization and generalization in practical training regimes where the optimal
learning rate is not very large, including small- to medium-batch training from
scratch on ImageNet and fine-tuning language models on downstream tasks.
