  In training neural networks, batch normalization has many benefits, not all
of them entirely understood. But it also has some drawbacks. Foremost is
arguably memory consumption, as computing the batch statistics requires all
instances within the batch to be processed simultaneously, whereas without
batch normalization it would be possible to process them one by one while
accumulating the weight gradients. Another drawback is that that distribution
parameters (mean and standard deviation) are unlike all other model parameters
in that they are not trained using gradient descent but require special
treatment, complicating implementation. In this paper, I show a simple and
straightforward way to address these issues. The idea, in short, is to add
terms to the loss that, for each activation, cause the minimization of the
negative log likelihood of a Gaussian distribution that is used to normalize
the activation. Among other benefits, this will hopefully contribute to the
democratization of AI research by means of lowering the hardware requirements
for training larger models.
