  While error backpropagation (BP) has dominated the training of nearly all
modern neural networks for a long time, it suffers from several biological
plausibility issues such as the symmetric weight requirement and synchronous
updates. Feedback Alignment (FA) was proposed as an alternative to BP to
address those dilemmas and has been demonstrated to be effective on various
tasks and network architectures. Despite its simplicity and effectiveness, a
satisfying explanation of how FA works across different architectures is still
lacking. Here we propose a novel, architecture-agnostic theory of how FA works
through the lens of information theory: Instead of approximating gradients
calculated by BP with the same parameter, FA learns effective representations
by embedding target information into neural networks to be trained. We show
this through the analysis of FA dynamics in idealized settings and then via a
series of experiments. Based on the implications of this theory, we designed
three variants of FA and show their comparable performance on several tasks.
These variants also account for some phenomena and theories in neuroscience
such as predictive coding and representational drift.
