  Learning in neural networks is often framed as a problem in which targeted
error signals are directly propagated to parameters and used to produce updates
that induce more optimal network behaviour. Backpropagation of error (BP) is an
example of such an approach and has proven to be a highly successful
application of stochastic gradient descent to deep neural networks. We propose
constrained parameter inference (COPI) as a new principle for learning. The
COPI approach assumes that learning can be set up in a manner where parameters
infer their own values based upon observations of their local neuron
activities. We find that this estimation of network parameters is possible
under the constraints of decorrelated neural inputs and top-down perturbations
of neural states for credit assignment. We show that the decorrelation required
for COPI allows learning at extremely high learning rates, competitive with
that of adaptive optimizers, as used by BP. We further demonstrate that COPI
affords a new approach to feature analysis and network compression. Finally, we
argue that COPI may shed new light on learning in biological networks given the
evidence for decorrelation in the brain.
