  In computational neuroscience, fixed points of recurrent neural networks are
commonly used to model neural responses to static or slowly changing stimuli.
These applications raise the question of how to train the weights in a
recurrent neural network to minimize a loss function evaluated on fixed points.
A natural approach is to use gradient descent on the Euclidean space of
synaptic weights. We show that this approach can lead to poor learning
performance due, in part, to singularities that arise in the loss surface. We
use a reparameterization of the recurrent network model to derive two
alternative learning rules that produces more robust learning dynamics. We show
that these learning rules can be interpreted as steepest descent and gradient
descent, respectively, under a non-Euclidean metric on the space of recurrent
weights. Our results question the common, implicit assumption that learning in
the brain should be expected to follow the negative Euclidean gradient of
synaptic weights.
