  This paper revisits the temporal difference (TD) learning algorithm for the
policy evaluation tasks in reinforcement learning. Typically, the performance
of TD(0) and TD($\lambda$) is very sensitive to the choice of stepsizes.
Oftentimes, TD(0) suffers from slow convergence. Motivated by the tight link
between the TD(0) learning algorithm and the stochastic gradient methods, we
develop a provably convergent adaptive projected variant of the TD(0) learning
algorithm with linear function approximation that we term AdaTD(0). In contrast
to the TD(0), AdaTD(0) is robust or less sensitive to the choice of stepsizes.
Analytically, we establish that to reach an $\epsilon$ accuracy, the number of
iterations needed is
$\tilde{O}(\epsilon^{-2}\ln^4\frac{1}{\epsilon}/\ln^4\frac{1}{\rho})$ in the
general case, where $\rho$ represents the speed of the underlying Markov chain
converges to the stationary distribution. This implies that the iteration
complexity of AdaTD(0) is no worse than that of TD(0) in the worst case. When
the stochastic semi-gradients are sparse, we provide theoretical acceleration
of AdaTD(0). Going beyond TD(0), we develop an adaptive variant of
TD($\lambda$), which is referred to as AdaTD($\lambda$). Empirically, we
evaluate the performance of AdaTD(0) and AdaTD($\lambda$) on several standard
reinforcement learning tasks, which demonstrate the effectiveness of our new
approaches.
