  Implementation of quantized neural networks on computing hardware leads to
considerable speed up and memory saving. However, quantized deep networks are
difficult to train and batch~normalization (BatchNorm) layer plays an important
role in training full-precision and quantized networks. Most studies on
BatchNorm are focused on full-precision networks, and there is little research
in understanding BatchNorm affect in quantized training which we address here.
We show BatchNorm avoids gradient explosion which is counter-intuitive and
recently observed in numerical experiments by other researchers.
