  To be effective in sequential data processing, Recurrent Neural Networks
(RNNs) are required to keep track of past events by creating memories. While
the relation between memories and the network's hidden state dynamics was
established over the last decade, previous works in this direction were of a
predominantly descriptive nature focusing mainly on locating the dynamical
objects of interest. In particular, it remained unclear how dynamical
observables affect the performance, how they form and whether they can be
manipulated. Here, we utilize different training protocols, datasets and
architectures to obtain a range of networks solving a delayed classification
task with similar performance, alongside substantial differences in their
ability to extrapolate for longer delays. We analyze the dynamics of the
network's hidden state, and uncover the reasons for this difference. Each
memory is found to be associated with a nearly steady state of the dynamics
which we refer to as a 'slow point'. Slow point speeds predict extrapolation
performance across all datasets, protocols and architectures tested.
Furthermore, by tracking the formation of the slow points we are able to
understand the origin of differences between training protocols. Finally, we
propose a novel regularization technique that is based on the relation between
hidden state speeds and memory longevity. Our technique manipulates these
speeds, thereby leading to a dramatic improvement in memory robustness over
time, and could pave the way for a new class of regularization methods.
