  A key component of most neural network architectures is the use of
normalization layers, such as Batch Normalization. Despite its common use and
large utility in optimizing deep architectures, it has been challenging both to
generically improve upon Batch Normalization and to understand the
circumstances that lend themselves to other enhancements. In this paper, we
identify four improvements to the generic form of Batch Normalization and the
circumstances under which they work, yielding performance gains across all
batch sizes while requiring no additional computation during training. These
contributions include proposing a method for reasoning about the current
example in inference normalization statistics, fixing a training vs. inference
discrepancy; recognizing and validating the powerful regularization effect of
Ghost Batch Normalization for small and medium batch sizes; examining the
effect of weight decay regularization on the scaling and shifting parameters
gamma and beta; and identifying a new normalization algorithm for very small
batch sizes by combining the strengths of Batch and Group Normalization. We
validate our results empirically on six datasets: CIFAR-100, SVHN, Caltech-256,
Oxford Flowers-102, CUB-2011, and ImageNet.
