  Online Normalization is a new technique for normalizing the hidden
activations of a neural network. Like Batch Normalization, it normalizes the
sample dimension. While Online Normalization does not use batches, it is as
accurate as Batch Normalization. We resolve a theoretical limitation of Batch
Normalization by introducing an unbiased technique for computing the gradient
of normalized activations. Online Normalization works with automatic
differentiation by adding statistical normalization as a primitive. This
technique can be used in cases not covered by some other normalizers, such as
recurrent networks, fully connected networks, and networks with activation
memory requirements prohibitive for batching. We show its applications to image
classification, image segmentation, and language modeling. We present formal
proofs and experimental results on ImageNet, CIFAR, and PTB datasets.
