  While there are convergence guarantees for temporal difference (TD) learning
when using linear function approximators, the situation for nonlinear models is
far less understood, and divergent examples are known. Here we take a first
step towards extending theoretical convergence guarantees to TD learning with
nonlinear function approximation. More precisely, we consider the expected
learning dynamics of the TD(0) algorithm for value estimation. As the step-size
converges to zero, these dynamics are defined by a nonlinear ODE which depends
on the geometry of the space of function approximators, the structure of the
underlying Markov chain, and their interaction. We find a set of function
approximators that includes ReLU networks and has geometry amenable to TD
learning regardless of environment, so that the solution performs about as well
as linear TD in the worst case. Then, we show how environments that are more
reversible induce dynamics that are better for TD learning and prove global
convergence to the true value function for well-conditioned function
approximators. Finally, we generalize a divergent counterexample to a family of
divergent problems to demonstrate how the interaction between approximator and
environment can go wrong and to motivate the assumptions needed to prove
convergence.
