  Task-based modeling with recurrent neural networks (RNNs) has emerged as a
popular way to infer the computational function of different brain regions.
These models are quantitatively assessed by comparing the low-dimensional
neural representations of the model with the brain, for example using canonical
correlation analysis (CCA). However, the nature of the detailed neurobiological
inferences one can draw from such efforts remains elusive. For example, to what
extent does training neural networks to solve common tasks uniquely determine
the network dynamics, independent of modeling architectural choices? Or
alternatively, are the learned dynamics highly sensitive to different model
choices? Knowing the answer to these questions has strong implications for
whether and how we should use task-based RNN modeling to understand brain
dynamics. To address these foundational questions, we study populations of
thousands of networks, with commonly used RNN architectures, trained to solve
neuroscientifically motivated tasks and characterize their nonlinear dynamics.
We find the geometry of the RNN representations can be highly sensitive to
different network architectures, yielding a cautionary tale for measures of
similarity that rely representational geometry, such as CCA. Moreover, we find
that while the geometry of neural dynamics can vary greatly across
architectures, the underlying computational scaffold---the topological
structure of fixed points, transitions between them, limit cycles, and
linearized dynamics---often appears universal across all architectures.
