  Learning relies on coordinated synaptic changes in recurrently connected
populations of neurons. Therefore, understanding the collective evolution of
synaptic connectivity over learning is a key challenge in neuroscience and
machine learning. In particular, recent work has shown that the weight matrices
of task-trained RNNs are typically low rank, but how this low rank structure
unfolds over learning is unknown. To address this, we investigate the rank of
the 3-tensor formed by the weight matrices throughout learning. By fitting RNNs
of varying rank to large-scale neural recordings during a motor learning task,
we find that the inferred weights are low-tensor-rank and therefore evolve over
a fixed low-dimensional subspace throughout the entire course of learning. We
next validate the observation of low-tensor-rank learning on an RNN trained to
solve the same task. Finally, we present a set of mathematical results bounding
the matrix and tensor ranks of gradient descent learning dynamics which show
that low-tensor-rank weights emerge naturally in RNNs trained to solve
low-dimensional tasks. Taken together, our findings provide insight on the
evolution of population connectivity over learning in both biological and
artificial neural networks, and enable reverse engineering of learning-induced
changes in recurrent dynamics from large-scale neural recordings.
