  The problem of quantizing the activations of a deep neural network is
considered. An examination of the popular binary quantization approach shows
that this consists of approximating a classical non-linearity, the hyperbolic
tangent, by two functions: a piecewise constant sign function, which is used in
feedforward network computations, and a piecewise linear hard tanh function,
used in the backpropagation step during network learning. The problem of
approximating the ReLU non-linearity, widely used in the recent deep learning
literature, is then considered. An half-wave Gaussian quantizer (HWGQ) is
proposed for forward approximation and shown to have efficient implementation,
by exploiting the statistics of of network activations and batch normalization
operations commonly used in the literature. To overcome the problem of gradient
mismatch, due to the use of different forward and backward approximations,
several piece-wise backward approximators are then investigated. The
implementation of the resulting quantized network, denoted as HWGQ-Net, is
shown to achieve much closer performance to full precision networks, such as
AlexNet, ResNet, GoogLeNet and VGG-Net, than previously available low-precision
networks, with 1-bit binary weights and 2-bit quantized activations.
