  A common optimization tool used in deep reinforcement learning is momentum,
which consists in accumulating and discounting past gradients, reapplying them
at each iteration. We argue that, unlike in supervised learning, momentum in
Temporal Difference (TD) learning accumulates gradients that become doubly
stale: not only does the gradient of the loss change due to parameter updates,
the loss itself changes due to bootstrapping. We first show that this
phenomenon exists, and then propose a first-order correction term to momentum.
We show that this correction term improves sample efficiency in policy
evaluation by correcting target value drift. An important insight of this work
is that deep RL methods are not always best served by directly importing
techniques from the supervised setting.
