  Herein the topics of (natural) gradient descent, data decorrelation, and
approximate methods for backpropagation are brought into a dialogue. Natural
gradient descent illuminates how gradient vectors, pointing at directions of
steepest descent, can be improved by considering the local curvature of loss
landscapes. We extend this perspective and show that to fully solve the problem
illuminated by natural gradients in neural networks, one must recognise that
correlations in the data at any linear transformation, including node responses
at every layer of a neural network, cause a non-orthonormal relationship
between the model's parameters. To solve this requires a solution to
decorrelate inputs at each individual layer of a neural network. We describe a
range of methods which have been proposed for decorrelation and whitening of
node output, while providing a novel method specifically useful for distributed
computing and computational neuroscience. Implementing decorrelation within
multi-layer neural networks, we can show that not only is training via
backpropagation sped up significantly but also existing approximations of
backpropagation, which have failed catastrophically in the past, are made
performant once more. This has the potential to provide a route forward for
approximate gradient descent methods which have previously been discarded,
training approaches for analogue and neuromorphic hardware, and potentially
insights as to the efficacy and utility of decorrelation processes in the
brain.
