  This paper studies accelerated algorithms for Q-learning. We propose an
acceleration scheme by incorporating the historical iterates of the Q-function.
The idea is conceptually inspired by the momentum-based acceleration methods in
the optimization theory. Under finite state-action space settings, the proposed
accelerated Q-learning algorithm provably converges to the global optimum with
a rate of $\mathcal{O}(1/\sqrt{T})$. While sharing a comparable theoretic
convergence rate with the existing Speedy Q-learning (SpeedyQ) algorithm, we
numerically show that the proposed algorithm outperforms SpeedyQ via playing
the FrozenLake grid world game. Furthermore, we generalize the acceleration
scheme to the continuous state-action space case where function approximation
of the Q-function is necessary. In this case, the algorithms are validated
using commonly adopted testing problems in reinforcement learning, including
two discrete-time linear quadratic regulation (LQR) problems from the Deepmind
Control Suite, and the Atari 2600 games. Simulation results show that the
proposed accelerated algorithms can improve the convergence performance
compared with the vanilla Q-learning algorithm.
