  We develop a theory of neural synaptic balance and how it can emerge or be
enforced in neural networks. For a given additive cost function $R$
(regularizer), a neuron is said to be in balance if the total cost of its input
weights is equal to the total cost of its output weights. The basic example is
provided by feedforward networks of ReLU units trained with $L_2$ regularizers,
which exhibit balance after proper training. The theory explains this
phenomenon and extends it in several directions. The first direction is the
extension to bilinear and other activation functions. The second direction is
the extension to more general regularizers, including all $L_p$ ($p>0$)
regularizers. The third direction is the extension to non-layered
architectures, recurrent architectures, convolutional architectures, as well as
architectures with mixed activation functions. The theory is based on two local
neuronal operations: scaling which is commutative, and balancing which is not
commutative. Finally, and most importantly, given any initial set of weights,
when local balancing operations are applied to each neuron in a stochastic
manner, global order always emerges through the convergence of the stochastic
balancing algorithm to the same unique set of balanced weights. The reason for
this convergence is the existence of an underlying strictly convex optimization
problem where the relevant variables are constrained to a linear, only
architecture-dependent, manifold. The theory is corroborated through various
simulations carried out on benchmark data sets. Scaling and balancing
operations are entirely local and thus physically plausible in biological and
neuromorphic networks.
