  Although adaptive gradient methods have been extensively used in deep
learning, their convergence rates proved in the literature are all slower than
that of SGD, particularly with respect to their dependence on the dimension.
This paper considers the classical RMSProp and its momentum extension and
establishes the convergence rate of $\frac{1}{T}\sum_{k=1}^T E\left[\|\nabla
f(x^k)\|_1\right]\leq O(\frac{\sqrt{d}C}{T^{1/4}})$ measured by $\ell_1$ norm
without the bounded gradient assumption, where $d$ is the dimension of the
optimization variable, $T$ is the iteration number, and $C$ is a constant
identical to that appeared in the optimal convergence rate of SGD. Our
convergence rate matches the lower bound with respect to all the coefficients
except the dimension $d$. Since $\|x\|_2\ll\|x\|_1\leq\sqrt{d}\|x\|_2$ for
problems with extremely large $d$, our convergence rate can be considered to be
analogous to the $\frac{1}{T}\sum_{k=1}^T E\left[\|\nabla f(x^k)\|_2\right]\leq
O(\frac{C}{T^{1/4}})$ rate of SGD in the ideal case of $\|\nabla
f(x)\|_1=\varTheta(\sqrt{d}\|\nabla f(x)\|_2)$.
