  Several recently proposed stochastic optimization methods that have been
successfully used in training deep networks such as RMSProp, Adam, Adadelta,
Nadam are based on using gradient updates scaled by square roots of exponential
moving averages of squared past gradients. In many applications, e.g. learning
with large output spaces, it has been empirically observed that these
algorithms fail to converge to an optimal solution (or a critical point in
nonconvex settings). We show that one cause for such failures is the
exponential moving average used in the algorithms. We provide an explicit
example of a simple convex optimization setting where Adam does not converge to
the optimal solution, and describe the precise problems with the previous
analysis of Adam algorithm. Our analysis suggests that the convergence issues
can be fixed by endowing such algorithms with `long-term memory' of past
gradients, and propose new variants of the Adam algorithm which not only fix
the convergence issues but often also lead to improved empirical performance.
