  The relation between neural activity and behaviorally relevant variables is
at the heart of neuroscience research. When strong, this relation is termed a
neural representation. There is increasing evidence, however, for partial
dissociations between activity in an area and relevant external variables.
While many explanations have been proposed, a theoretical framework for the
relationship between external and internal variables is lacking. Here, we
utilize recurrent neural networks (RNNs) to explore the question of when and
how neural dynamics and the network's output are related from a geometrical
point of view. We find that training RNNs can lead to two dynamical regimes:
dynamics can either be aligned with the directions that generate output
variables, or oblique to them. We show that the choice of readout weight
magnitude before training can serve as a control knob between the regimes,
similar to recent findings in feedforward networks. These regimes are
functionally distinct. Oblique networks are more heterogeneous and suppress
noise in their output directions. They are furthermore more robust to
perturbations along the output directions. Crucially, the oblique regime is
specific to recurrent (but not feedforward) networks, arising from dynamical
stability considerations. Finally, we show that tendencies towards the aligned
or the oblique regime can be dissociated in neural recordings. Altogether, our
results open a new perspective for interpreting neural activity by relating
network dynamics and their output.
