  Neural dynamical systems with stable attractor structures, such as point
attractors and continuous attractors, are hypothesized to underlie meaningful
temporal behavior that requires working memory. However, working memory may not
support useful learning signals necessary to adapt to changes in the temporal
structure of the environment. We show that in addition to the continuous
attractors that are widely implicated, periodic and quasi-periodic attractors
can also support learning arbitrarily long temporal relationships. Unlike the
continuous attractors that suffer from the fine-tuning problem, the less
explored quasi-periodic attractors are uniquely qualified for learning to
produce temporally structured behavior. Our theory has broad implications for
the design of artificial learning systems and makes predictions about
observable signatures of biological neural dynamics that can support temporal
dependence learning and working memory. Based on our theory, we developed a new
initialization scheme for artificial recurrent neural networks that outperforms
standard methods for tasks that require learning temporal dynamics. Moreover,
we propose a robust recurrent memory mechanism for integrating and maintaining
head direction without a ring attractor.
