  In this paper, we provide a rigorous proof of convergence of the Adaptive
Moment Estimate (Adam) algorithm for a wide class of optimization objectives.
Despite the popularity and efficiency of the Adam algorithm in training deep
neural networks, its theoretical properties are not yet fully understood, and
existing convergence proofs require unrealistically strong assumptions, such as
globally bounded gradients, to show the convergence to stationary points. In
this paper, we show that Adam provably converges to $\epsilon$-stationary
points with ${O}(\epsilon^{-4})$ gradient complexity under far more realistic
conditions. The key to our analysis is a new proof of boundedness of gradients
along the optimization trajectory of Adam, under a generalized smoothness
assumption according to which the local smoothness (i.e., Hessian norm when it
exists) is bounded by a sub-quadratic function of the gradient norm. Moreover,
we propose a variance-reduced version of Adam with an accelerated gradient
complexity of ${O}(\epsilon^{-3})$.
