  Adaptive gradient methods, e.g. \textsc{Adam}, have achieved tremendous
success in machine learning. Scaling the learning rate element-wisely by a
certain form of second moment estimate of gradients, such methods are able to
attain rapid training of modern deep neural networks. Nevertheless, they are
observed to suffer from compromised generalization ability compared with
stochastic gradient descent (\textsc{SGD}) and tend to be trapped in local
minima at an early stage during training. Intriguingly, we discover that
substituting the gradient in the second raw moment estimate term with its
momentumized version in \textsc{Adam} can resolve the issue. The intuition is
that gradient with momentum contains more accurate directional information and
therefore its second moment estimation is a more favorable option for learning
rate scaling than that of the raw gradient. Thereby we propose
\textsc{AdaMomentum} as a new optimizer reaching the goal of training fast
while generalizing much better. We further develop a theory to back up the
improvement in generalization and provide convergence guarantees under both
convex and nonconvex settings. Extensive experiments on a wide range of tasks
and models demonstrate that \textsc{AdaMomentum} exhibits state-of-the-art
performance and superior training stability consistently.
