  Deep Neural Networks (DNNs) thrive in recent years in which Batch
Normalization (BN) plays an indispensable role. However, it has been observed
that BN is costly due to the reduction operations. In this paper, we propose
alleviating this problem through sampling only a small fraction of data for
normalization at each iteration. Specifically, we model it as a statistical
sampling problem and identify that by sampling less correlated data, we can
largely reduce the requirement of the number of data for statistics estimation
in BN, which directly simplifies the reduction operations. Based on this
conclusion, we propose two sampling strategies, "Batch Sampling" (randomly
select several samples from each batch) and "Feature Sampling" (randomly select
a small patch from each feature map of all samples), that take both
computational efficiency and sample correlation into consideration.
Furthermore, we introduce an extremely simple variant of BN, termed as Virtual
Dataset Normalization (VDN), that can normalize the activations well with few
synthetical random samples. All the proposed methods are evaluated on various
datasets and networks, where an overall training speedup by up to 20% on GPU is
practically achieved without the support of any specialized libraries, and the
loss on accuracy and convergence rate are negligible. Finally, we extend our
work to the "micro-batch normalization" problem and yield comparable
performance with existing approaches at the case of tiny batch size.
