  Training neural networks is an optimization problem, and finding a decent set
of parameters through gradient descent can be a difficult task. A host of
techniques has been developed to aid this process before and during the
training phase. One of the most important and widely used class of method is
normalization. It is generally favorable for neurons to receive inputs that are
distributed with zero mean and unit variance, so we use statistics about
dataset to normalize them before the first layer. However, this property cannot
be guaranteed for the intermediate activations inside the network. A widely
used method to enforce this property inside the network is batch normalization.
It was developed to combat covariate shift inside networks. Empirically it is
known to work, but there is a lack of theoretical understanding about its
effectiveness and potential drawbacks it might have when used in practice. This
work studies batch normalization in detail, while comparing it with other
methods such as weight normalization, gradient clipping and dropout. The main
purpose of this work is to determine if it is possible to train networks
effectively when batch normalization is removed through adaption of the
training process.
