  Backpropagation (BP) remains the dominant and most successful method for
training parameters of deep neural network models. However, BP relies on two
computationally distinct phases, does not provide a satisfactory explanation of
biological learning, and can be challenging to apply for training of networks
with discontinuities or noisy node dynamics. By comparison, node perturbation
(NP) proposes learning by the injection of noise into network activations, and
subsequent measurement of the induced loss change. NP relies on two forward
(inference) passes, does not make use of network derivatives, and has been
proposed as a model for learning in biological systems. However, standard NP is
highly data inefficient and unstable due to its unguided noise-based search
process. In this work, we investigate different formulations of NP and relate
it to the concept of directional derivatives as well as combining it with a
decorrelating mechanism for layer-wise inputs. We find that a closer alignment
with directional derivatives together with input decorrelation at every layer
strongly enhances performance of NP learning with large improvements in
parameter convergence and much higher performance on the test data, approaching
that of BP. Furthermore, our novel formulation allows for application to noisy
systems in which the noise process itself is inaccessible.
