  We propose and analyze an alternate approach to off-policy multi-step
temporal difference learning, in which off-policy returns are corrected with
the current Q-function in terms of rewards, rather than with the target policy
in terms of transition probabilities. We prove that such approximate
corrections are sufficient for off-policy convergence both in policy evaluation
and control, provided certain conditions. These conditions relate the distance
between the target and behavior policies, the eligibility trace parameter and
the discount factor, and formalize an underlying tradeoff in off-policy
TD($\lambda$). We illustrate this theoretical relationship empirically on a
continuous-state control task.
