  We introduce the "NoBackTrack" algorithm to train the parameters of dynamical
systems such as recurrent neural networks. This algorithm works in an online,
memoryless setting, thus requiring no backpropagation through time, and is
scalable, avoiding the large computational and memory cost of maintaining the
full gradient of the current state with respect to the parameters.
  The algorithm essentially maintains, at each time, a single search direction
in parameter space. The evolution of this search direction is partly stochastic
and is constructed in such a way to provide, at every time, an unbiased random
estimate of the gradient of the loss function with respect to the parameters.
Because the gradient estimate is unbiased, on average over time the parameter
is updated as it should.
  The resulting gradient estimate can then be fed to a lightweight Kalman-like
filter to yield an improved algorithm. For recurrent neural networks, the
resulting algorithms scale linearly with the number of parameters.
  Small-scale experiments confirm the suitability of the approach, showing that
the stochastic approximation of the gradient introduced in the algorithm is not
detrimental to learning. In particular, the Kalman-like version of NoBackTrack
is superior to backpropagation through time (BPTT) when the time span of
dependencies in the data is longer than the truncation span for BPTT.
