  In this paper, we have studied how the training of the convolutional neural
networks (CNNs) can be affected by changing the position of the batch
normalization (BN) layer. Three different convolutional neural networks have
been chosen for our experiments. These networks are AlexNet, VGG-16, and
ResNet- 20. We show that the speed up in training provided by the BN algorithm
can be improved by using other positions for the BN layer than the one
suggested by its original paper. Also, we discuss how the BN layer in a certain
position can aid the training of one network but not the other. Three different
positions for the BN layer have been studied in this research. These positions
are: the BN layer between the convolution layer and the non-linear activation
function, the BN layer after the non-linear activation function and finally,
the BN layer before each of the convolutional layers.
