  It is still common to use Q-learning and temporal difference (TD)
learning-even though they have divergence issues and sound Gradient TD
alternatives exist-because divergence seems rare and they typically perform
well. However, recent work with large neural network learning systems reveals
that instability is more common than previously thought. Practitioners face a
difficult dilemma: choose an easy to use and performant TD method, or a more
complex algorithm that is more sound but harder to tune and all but unexplored
with non-linear function approximation or control. In this paper, we introduce
a new method called TD with Regularized Corrections (TDRC), that attempts to
balance ease of use, soundness, and performance. It behaves as well as TD, when
TD performs well, but is sound in cases where TD diverges. We empirically
investigate TDRC across a range of problems, for both prediction and control,
and for both linear and non-linear function approximation, and show,
potentially for the first time, that gradient TD methods could be a better
alternative to TD and Q-learning.
