  Low-bit deep neural networks (DNNs) become critical for embedded applications
due to their low storage requirement and computing efficiency. However, they
suffer much from the non-negligible accuracy drop. This paper proposes the
stochastic quantization (SQ) algorithm for learning accurate low-bit DNNs. The
motivation is due to the following observation. Existing training algorithms
approximate the real-valued elements/filters with low-bit representation all
together in each iteration. The quantization errors may be small for some
elements/filters, while are remarkable for others, which lead to inappropriate
gradient direction during training, and thus bring notable accuracy drop.
Instead, SQ quantizes a portion of elements/filters to low-bit with a
stochastic probability inversely proportional to the quantization error, while
keeping the other portion unchanged with full-precision. The quantized and
full-precision portions are updated with corresponding gradients separately in
each iteration. The SQ ratio is gradually increased until the whole network is
quantized. This procedure can greatly compensate the quantization error and
thus yield better accuracy for low-bit DNNs. Experiments show that SQ can
consistently and significantly improve the accuracy for different low-bit DNNs
on various datasets and various network structures.
