  Batch normalization (BN) has become a de facto standard for training deep
convolutional networks. However, BN accounts for a significant fraction of
training run-time and is difficult to accelerate, since it is a
memory-bandwidth bounded operation. Such a drawback of BN motivates us to
explore recently proposed weight normalization algorithms (WN algorithms), i.e.
weight normalization, normalization propagation and weight normalization with
translated ReLU. These algorithms don't slow-down training iterations and were
experimentally shown to outperform BN on relatively small networks and
datasets. However, it is not clear if these algorithms could replace BN in
practical, large-scale applications. We answer this question by providing a
detailed comparison of BN and WN algorithms using ResNet-50 network trained on
ImageNet. We found that although WN achieves better training accuracy, the
final test accuracy is significantly lower ($\approx 6\%$) than that of BN.
This result demonstrates the surprising strength of the BN regularization
effect which we were unable to compensate for using standard regularization
techniques like dropout and weight decay. We also found that training of deep
networks with WN algorithms is significantly less stable compared to BN,
limiting their practical applications.
