  Integrating adaptive learning rate and momentum techniques into SGD leads to
a large class of efficiently accelerated adaptive stochastic algorithms, such
as AdaGrad, RMSProp, Adam, AccAdaGrad, \textit{etc}. In spite of their
effectiveness in practice, there is still a large gap in their theories of
convergences, especially in the difficult non-convex stochastic setting. To
fill this gap, we propose \emph{weighted AdaGrad with unified momentum}, dubbed
AdaUSM, which has the main characteristics that (1) it incorporates a unified
momentum scheme which covers both the heavy ball momentum and the Nesterov
accelerated gradient momentum; (2) it adopts a novel weighted adaptive learning
rate that can unify the learning rates of AdaGrad, AccAdaGrad, Adam, and
RMSProp. Moreover, when we take polynomially growing weights in AdaUSM, we
obtain its $\mathcal{O}(\log(T)/\sqrt{T})$ convergence rate in the non-convex
stochastic setting. We also show that the adaptive learning rates of Adam and
RMSProp correspond to taking exponentially growing weights in AdaUSM, thereby
providing a new perspective for understanding Adam and RMSProp. Lastly,
comparative experiments of AdaUSM against SGD with momentum, AdaGrad, AdaEMA,
Adam, and AMSGrad on various deep learning models and datasets are also carried
out.
