  Recurrent neural networks (RNNs) are popular machine learning tools for
modeling and forecasting sequential data and for inferring dynamical systems
(DS) from observed time series. Concepts from DS theory (DST) have variously
been used to further our understanding of both, how trained RNNs solve complex
tasks, and the training process itself. Bifurcations are particularly important
phenomena in DS, including RNNs, that refer to topological (qualitative)
changes in a system's dynamical behavior as one or more of its parameters are
varied. Knowing the bifurcation structure of an RNN will thus allow to deduce
many of its computational and dynamical properties, like its sensitivity to
parameter variations or its behavior during training. In particular,
bifurcations may account for sudden loss jumps observed in RNN training that
could severely impede the training process. Here we first mathematically prove
for a particular class of ReLU-based RNNs that certain bifurcations are indeed
associated with loss gradients tending toward infinity or zero. We then
introduce a novel heuristic algorithm for detecting all fixed points and
k-cycles in ReLU-based RNNs and their existence and stability regions, hence
bifurcation manifolds in parameter space. In contrast to previous numerical
algorithms for finding fixed points and common continuation methods, our
algorithm provides exact results and returns fixed points and cycles up to high
orders with surprisingly good scaling behavior. We exemplify the algorithm on
the analysis of the training process of RNNs, and find that the recently
introduced technique of generalized teacher forcing completely avoids certain
types of bifurcations in training. Thus, besides facilitating the DST analysis
of trained RNNs, our algorithm provides a powerful instrument for analyzing the
training process itself.
