  Motivated by applications in reinforcement learning (RL), we study a
nonlinear stochastic approximation (SA) algorithm under Markovian noise, and
establish its finite-sample convergence bounds under various stepsizes.
Specifically, we show that when using constant stepsize (i.e., $\alpha_k\equiv
\alpha$), the algorithm achieves exponential fast convergence to a neighborhood
(with radius $O(\alpha\log(1/\alpha))$) around the desired limit point. When
using diminishing stepsizes with appropriate decay rate, the algorithm
converges with rate $O(\log(k)/k)$. Our proof is based on Lyapunov drift
arguments, and to handle the Markovian noise, we exploit the fast mixing of the
underlying Markov chain.
  To demonstrate the generality of our theoretical results on Markovian SA, we
use it to derive the finite-sample bounds of the popular $Q$-learning with
linear function approximation algorithm, under a condition on the behavior
policy. Importantly, we do not need to make the assumption that the samples are
i.i.d., and do not require an artificial projection step in the algorithm to
maintain the boundedness of the iterates. Numerical simulations corroborate our
theoretical results.
