  Batch Normalization (BN) is widely used to stabilize the optimization process
and improve the test performance of deep neural networks. The regularization
effect of BN depends on the batch size and explicitly using smaller batch sizes
with Batch Normalization, a method known as Ghost Batch Normalization (GBN),
has been found to improve generalization in many settings. We investigate the
effectiveness of GBN by disentangling the induced ``Ghost Noise'' from
normalization and quantitatively analyzing the distribution of noise as well as
its impact on model performance. Inspired by our analysis, we propose a new
regularization technique called Ghost Noise Injection (GNI) that imitates the
noise in GBN without incurring the detrimental train-test discrepancy effects
of small batch training. We experimentally show that GNI can provide a greater
generalization benefit than GBN. Ghost Noise Injection can also be beneficial
in otherwise non-noisy settings such as layer-normalized networks, providing
additional evidence of the usefulness of Ghost Noise in Batch Normalization as
a regularizer.
