  Despite the wide applications of Adam in reinforcement learning (RL), the
theoretical convergence of Adam-type RL algorithms has not been established.
This paper provides the first such convergence analysis for two fundamental RL
algorithms of policy gradient (PG) and temporal difference (TD) learning that
incorporate AMSGrad updates (a standard alternative of Adam in theoretical
analysis), referred to as PG-AMSGrad and TD-AMSGrad, respectively. Moreover,
our analysis focuses on Markovian sampling for both algorithms. We show that
under general nonlinear function approximation, PG-AMSGrad with a constant
stepsize converges to a neighborhood of a stationary point at the rate of
$\mathcal{O}(1/T)$ (where $T$ denotes the number of iterations), and with a
diminishing stepsize converges exactly to a stationary point at the rate of
$\mathcal{O}(\log^2 T/\sqrt{T})$. Furthermore, under linear function
approximation, TD-AMSGrad with a constant stepsize converges to a neighborhood
of the global optimum at the rate of $\mathcal{O}(1/T)$, and with a diminishing
stepsize converges exactly to the global optimum at the rate of
$\mathcal{O}(\log T/\sqrt{T})$. Our study develops new techniques for analyzing
the Adam-type RL algorithms under Markovian sampling.
