  We present a simple linear regression based approach for learning the weights
and biases of a neural network, as an alternative to standard gradient based
backpropagation. The present work is exploratory in nature, and we restrict the
description and experiments to (i) simple feedforward neural networks, (ii)
scalar (single output) regression problems, and (iii) invertible activation
functions. However, the approach is intended to be extensible to larger, more
complex architectures. The key idea is the observation that the input to every
neuron in a neural network is a linear combination of the activations of
neurons in the previous layer, as well as the parameters (weights and biases)
of the layer. If we are able to compute the ideal total input values to every
neuron by working backwards from the output, we can formulate the learning
problem as a linear least squares problem which iterates between updating the
parameters and the activation values. We present an explicit algorithm that
implements this idea, and we show that (at least for small problems) the
approach is more stable and faster than gradient-based methods.
