  The ubiquitous backpropagation algorithm requires sequential updates through
the network introducing a locking problem. In addition, back-propagation relies
on the transpose of forward weight matrices to compute updates, introducing a
weight transport problem across the network. Locking and weight transport are
problems because they prevent efficient parallelization and horizontal scaling
of the training process. We propose a new method to address both these problems
and scale up the training of large models. Our method works by dividing a deep
neural network into blocks and introduces a feedback network that propagates
the information from the targets backwards to provide auxiliary local losses.
Forward and backward propagation can operate in parallel and with different
sets of weights, addressing the problems of locking and weight transport. Our
approach derives from a statistical interpretation of training that treats
output activations of network blocks as parameters of probability
distributions. The resulting learning framework uses these parameters to
evaluate the agreement between forward and backward information. Error
backpropagation is then performed locally within each block, leading to
"block-local" learning. Several previously proposed alternatives to error
backpropagation emerge as special cases of our model. We present results on a
variety of tasks and architectures, demonstrating state-of-the-art performance
using block-local learning. These results provide a new principled framework
for training networks in a distributed setting.
