  Batch Normalization (BN) is extensively employed in various network
architectures by performing standardization within mini-batches.
  A full understanding of the process has been a central target in the deep
learning communities.
  Unlike existing works, which usually only analyze the standardization
operation, this paper investigates the more general Batch Whitening (BW). Our
work originates from the observation that while various whitening
transformations equivalently improve the conditioning, they show significantly
different behaviors in discriminative scenarios and training Generative
Adversarial Networks (GANs).
  We attribute this phenomenon to the stochasticity that BW introduces.
  We quantitatively investigate the stochasticity of different whitening
transformations and show that it correlates well with the optimization
behaviors during training.
  We also investigate how stochasticity relates to the estimation of population
statistics during inference.
  Based on our analysis, we provide a framework for designing and comparing BW
algorithms in different scenarios.
  Our proposed BW algorithm improves the residual networks by a significant
margin on ImageNet classification.
  Besides, we show that the stochasticity of BW can improve the GAN's
performance with, however, the sacrifice of the training stability.
