  Batch Normalization (BN) techniques have been proposed to reduce the
so-called Internal Covariate Shift (ICS) by attempting to keep the
distributions of layer outputs unchanged. Experiments have shown their
effectiveness on training deep neural networks. However, since only the first
two moments are controlled in these BN techniques, it seems that a weak
constraint is imposed on layer distributions and furthermore whether such
constraint can reduce ICS is unknown. Thus this paper proposes a measure for
ICS by using the Earth Mover (EM) distance and then derives the upper and lower
bounds for the measure to provide a theoretical analysis of BN. The upper bound
has shown that BN techniques can control ICS only for the outputs with low
dimensions and small noise whereas their control is NOT effective in other
cases. This paper also proves that such control is just a bounding of ICS
rather than a reduction of ICS. Meanwhile, the analysis shows that the
high-order moments and noise, which BN cannot control, have great impact on the
lower bound. Based on such analysis, this paper furthermore proposes an
algorithm that unitizes the outputs with an adjustable parameter to further
bound ICS in order to cope with the problems of BN. The upper bound for the
proposed unitization is noise-free and only dominated by the parameter. Thus,
the parameter can be trained to tune the bound and further to control ICS.
Besides, the unitization is embedded into the framework of BN to reduce the
information loss. The experiments show that this proposed algorithm outperforms
existing BN techniques on CIFAR-10, CIFAR-100 and ImageNet datasets.
