  Momentum is a widely used technique for gradient-based optimizers in deep
learning. In this paper, we propose a decaying momentum (\textsc{Demon}) rule.
We conduct the first large-scale empirical analysis of momentum decay methods
for modern neural network optimization, in addition to the most popular
learning rate decay schedules. Across 28 relevant combinations of models,
epochs, datasets, and optimizers, \textsc{Demon} achieves the highest number of
Top-1 and Top-3 finishes at 39\% and 85\% respectively, almost doubling the
second-placed learning rate cosine schedule at 17\% and 60\%, respectively.
\textsc{Demon} also outperforms other widely used schedulers including, but not
limited to, the learning rate step schedule, linear schedule, OneCycle
schedule, and exponential schedule. Compared with the widely used learning rate
step schedule, \textsc{Demon} is observed to be less sensitive to parameter
tuning, which is critical to training neural networks in practice. Results are
demonstrated across a variety of settings and architectures, including image
classification, generative models, and language models. \textsc{Demon} is easy
to implement, requires no additional tuning, and incurs almost no extra
computational overhead compared to the vanilla counterparts. Code is readily
available.
