  Feedback alignment algorithms are an alternative to backpropagation to train
neural networks, whereby some of the partial derivatives that are required to
compute the gradient are replaced by random terms. This essentially transforms
the update rule into a random walk in weight space. Surprisingly, learning
still works with those algorithms, including training of deep neural networks.
This is generally attributed to an alignment of the update of the random walker
with the true gradient - the eponymous gradient alignment -- which drives an
approximate gradient descend. The mechanism that leads to this alignment
remains unclear, however. In this paper, we use mathematical reasoning and
simulations to investigate gradient alignment. We observe that the feedback
alignment update rule has fixed points, which correspond to extrema of the loss
function. We show that gradient alignment is a stability criterion for those
fixed points. It is only a necessary criterion for algorithm performance.
Experimentally, we demonstrate that high levels of gradient alignment can lead
to poor algorithm performance and that the alignment is not always driving the
gradient descend.
