  Multi-step temporal difference (TD) learning is an important approach in
reinforcement learning, as it unifies one-step TD learning with Monte Carlo
methods in a way where intermediate algorithms can outperform either extreme.
They address a bias-variance trade off between reliance on current estimates,
which could be poor, and incorporating longer sampled reward sequences into the
updates. Especially in the off-policy setting, where the agent aims to learn
about a policy different from the one generating its behaviour, the variance in
the updates can cause learning to diverge as the number of sampled rewards used
in the estimates increases. In this paper, we introduce per-decision control
variates for multi-step TD algorithms, and compare them to existing methods.
Our results show that including the control variates can greatly improve
performance on both on and off-policy multi-step temporal difference learning
tasks.
