  This study introduces a new normalization layer termed Batch Layer
Normalization (BLN) to reduce the problem of internal covariate shift in deep
neural network layers. As a combined version of batch and layer normalization,
BLN adaptively puts appropriate weight on mini-batch and feature normalization
based on the inverse size of mini-batches to normalize the input to a layer
during the learning process. It also performs the exact computation with a
minor change at inference times, using either mini-batch statistics or
population statistics. The decision process to either use statistics of
mini-batch or population gives BLN the ability to play a comprehensive role in
the hyper-parameter optimization process of models. The key advantage of BLN is
the support of the theoretical analysis of being independent of the input data,
and its statistical configuration heavily depends on the task performed, the
amount of training data, and the size of batches. Test results indicate the
application potential of BLN and its faster convergence than batch
normalization and layer normalization in both Convolutional and Recurrent
Neural Networks. The code of the experiments is publicly available online
(https://github.com/A2Amir/Batch-Layer-Normalization).
