  Batch normalization (BN) allows training very deep networks by normalizing
activations by mini-batch sample statistics which renders BN unstable for small
batch sizes. Current small-batch solutions such as Instance Norm, Layer Norm,
and Group Norm use channel statistics which can be computed even for a single
sample. Such methods are less stable than BN as they critically depend on the
statistics of a single input sample. To address this problem, we propose a
normalization of activation without sample statistics. We present WeightAlign:
a method that normalizes the weights by the mean and scaled standard derivation
computed within a filter, which normalizes activations without computing any
sample statistics. Our proposed method is independent of batch size and stable
over a wide range of batch sizes. Because weight statistics are orthogonal to
sample statistics, we can directly combine WeightAlign with any method for
activation normalization. We experimentally demonstrate these benefits for
classification on CIFAR-10, CIFAR-100, ImageNet, for semantic segmentation on
PASCAL VOC 2012 and for domain adaptation on Office-31.
