  Batch Normalization (BN) is a core and prevalent technique in accelerating
the training of deep neural networks and improving the generalization on
Computer Vision (CV) tasks. However, it fails to defend its position in Natural
Language Processing (NLP), which is dominated by Layer Normalization (LN). In
this paper, we are trying to answer why BN usually performs worse than LN in
NLP tasks with Transformer models. We find that the inconsistency between
training and inference of BN is the leading cause that results in the failure
of BN in NLP. We define Training Inference Discrepancy (TID) to quantitatively
measure this inconsistency and reveal that TID can indicate BN's performance,
supported by extensive experiments, including image classification, neural
machine translation, language modeling, sequence labeling, and text
classification tasks. We find that BN can obtain much better test performance
than LN when TID keeps small through training. To suppress the explosion of
TID, we propose Regularized BN (RBN) that adds a simple regularization term to
narrow the gap between batch statistics and population statistics of BN. RBN
improves the performance of BN consistently and outperforms or is on par with
LN on 17 out of 20 settings, involving ten datasets and two common variants of
Transformer
  Our code is available at https://github.com/wjxts/RegularizedBN.
