  Network quantization generally converts full-precision weights and/or
activations into low-bit fixed-point values in order to accelerate an inference
process. Recent approaches to network quantization further discretize the
gradients into low-bit fixed-point values, enabling an efficient training. They
typically set a quantization interval using a min-max range of the gradients or
adjust the interval such that the quantization error for entire gradients is
minimized. In this paper, we analyze the quantization error of gradients for
the low-bit fixed-point training, and show that lowering the error for
large-magnitude gradients boosts the quantization performance significantly.
Based on this, we derive an upper bound of quantization error for the large
gradients in terms of the quantization interval, and obtain an optimal
condition for the interval minimizing the quantization error for large
gradients. We also introduce an interval update algorithm that adjusts the
quantization interval adaptively to maintain a small quantization error for
large gradients. Experimental results demonstrate the effectiveness of our
quantization method for various combinations of network architectures and
bit-widths on various tasks, including image classification, object detection,
and super-resolution.
