  Deep neural networks rely heavily on normalization methods to improve their
performance and learning behavior. Although normalization methods spurred the
development of increasingly deep and efficient architectures, they also
increase the vulnerability with respect to noise and input corruptions. In most
applications, however, noise is ubiquitous and diverse; this can often lead to
complete failure of machine learning systems as they fail to cope with
mismatches between the input distribution during training- and test-time. The
most common normalization method, batch normalization, reduces the distribution
shift during training but is agnostic to changes in the input distribution
during test time. This makes batch normalization prone to performance
degradation whenever noise is present during test-time. Sample-based
normalization methods can correct linear transformations of the activation
distribution but cannot mitigate changes in the distribution shape; this makes
the network vulnerable to distribution changes that cannot be reflected in the
normalization parameters. We propose an unsupervised non-parametric
distribution correction method that adapts the activation distribution of each
layer. This reduces the mismatch between the training and test-time
distribution by minimizing the 1-D Wasserstein distance. In our experiments, we
empirically show that the proposed method effectively reduces the impact of
intense image corruptions and thus improves the classification performance
without the need for retraining or fine-tuning the model.
