  In this paper, we analyze batch normalization from the perspective of
discriminability and find the disadvantages ignored by previous studies: the
difference in $l_2$ norms of sample features can hinder batch normalization
from obtaining more distinguished inter-class features and more compact
intra-class features. To address this issue, we propose a simple yet effective
method to equalize the $l_2$ norms of sample features. Concretely, we
$l_2$-normalize each sample feature before feeding them into batch
normalization, and therefore the features are of the same magnitude. Since the
proposed method combines the $l_2$ normalization and batch normalization, we
name our method $L_2$BN. The $L_2$BN can strengthen the compactness of
intra-class features and enlarge the discrepancy of inter-class features. The
$L_2$BN is easy to implement and can exert its effect without any additional
parameters or hyper-parameters. We evaluate the effectiveness of $L_2$BN
through extensive experiments with various models on image classification and
acoustic scene classification tasks. The results demonstrate that the $L_2$BN
can boost the generalization ability of various neural network models and
achieve considerable performance improvements.
