  This paper studies accelerations in Q-learning algorithms. We propose an
accelerated target update scheme by incorporating the historical iterates of Q
functions. The idea is conceptually inspired by the momentum-based accelerated
methods in the optimization theory. Conditions under which the proposed
accelerated algorithms converge are established. The algorithms are validated
using commonly adopted testing problems in reinforcement learning, including
the FrozenLake grid world game, two discrete-time LQR problems from the
Deepmind Control Suite, and the Atari 2600 games. Simulation results show that
the proposed accelerated algorithms can improve the convergence performance
compared with the vanilla Q-learning algorithm.
