Effective Multi-step Temporal-Difference Learning for Non-Linear
  Function Approximation
  Multi-step temporal-difference (TD) learning, where the update targets
contain information from multiple time steps ahead, is one of the most popular
forms of TD learning for linear function approximation. The reason is that
multi-step methods often yield substantially better performance than their
single-step counter-parts, due to a lower bias of the update targets. For
non-linear function approximation, however, single-step methods appear to be
the norm. Part of the reason could be that on many domains the popular
multi-step methods TD($\lambda$) and Sarsa($\lambda$) do not perform well when
combined with non-linear function approximation. In particular, they are very
susceptible to divergence of value estimates. In this paper, we identify the
reason behind this. Furthermore, based on our analysis, we propose a new
multi-step TD method for non-linear function approximation that addresses this
issue. We confirm the effectiveness of our method using two benchmark tasks
with neural networks as function approximation.
