  Integral to recent successes in deep reinforcement learning has been a class
of temporal difference methods that use infrequently updated target values for
policy evaluation in a Markov Decision Process. Yet a complete theoretical
explanation for the effectiveness of target networks remains elusive. In this
work, we provide an analysis of this popular class of algorithms, to finally
answer the question: `why do target networks stabilise TD learning'? To do so,
we formalise the notion of a partially fitted policy evaluation method, which
describes the use of target networks and bridges the gap between fitted methods
and semigradient temporal difference algorithms. Using this framework we are
able to uniquely characterise the so-called deadly triad - the use of TD
updates with (nonlinear) function approximation and off-policy data - which
often leads to nonconvergent algorithms. This insight leads us to conclude that
the use of target networks can mitigate the effects of poor conditioning in the
Jacobian of the TD update. Instead, we show that under mild regularity
conditions and a well tuned target network update frequency, convergence can be
guaranteed even in the extremely challenging off-policy sampling and nonlinear
function approximation setting.
