  Back-propagation is a popular machine learning algorithm that uses gradient
descent in training neural networks for supervised learning, but can be very
slow. A number of algorithms have been developed to speed up convergence and
improve robustness of the learning. However, they are complicated to implement
biologically as they require information from previous updates. Inspired by
synaptic competition in biology, we have come up with a simple and local
gradient descent optimization algorithm that can reduce training time, with no
demand on past details. Our algorithm, named dynamic learning rate (DLR), works
similarly to the traditional gradient descent used in back-propagation, except
that instead of having a uniform learning rate across all synapses, the
learning rate depends on the current neuronal connection weights. Our algorithm
is found to speed up learning, particularly for small networks.
