  Regularization and normalization have become indispensable components in
training deep neural networks, resulting in faster training and improved
generalization performance. We propose the projected error function
regularization loss (PER) that encourages activations to follow the standard
normal distribution. PER randomly projects activations onto one-dimensional
space and computes the regularization loss in the projected space. PER is
similar to the Pseudo-Huber loss in the projected space, thus taking advantage
of both $L^1$ and $L^2$ regularization losses. Besides, PER can capture the
interaction between hidden units by projection vector drawn from a unit sphere.
By doing so, PER minimizes the upper bound of the Wasserstein distance of order
one between an empirical distribution of activations and the standard normal
distribution. To the best of the authors' knowledge, this is the first work to
regularize activations via distribution matching in the probability
distribution space. We evaluate the proposed method on the image classification
task and the word-level language modeling task.
