  Recurrent neural networks have achieved excellent performance in many
applications. However, on portable devices with limited resources, the models
are often too large to deploy. For applications on the server with large scale
concurrent requests, the latency during inference can also be very critical for
costly computing resources. In this work, we address these problems by
quantizing the network, both weights and activations, into multiple binary
codes {-1,+1}. We formulate the quantization as an optimization problem. Under
the key observation that once the quantization coefficients are fixed the
binary codes can be derived efficiently by binary search tree, alternating
minimization is then applied. We test the quantization for two well-known RNNs,
i.e., long short term memory (LSTM) and gated recurrent unit (GRU), on the
language models. Compared with the full-precision counter part, by 2-bit
quantization we can achieve ~16x memory saving and ~6x real inference
acceleration on CPUs, with only a reasonable loss in the accuracy. By 3-bit
quantization, we can achieve almost no loss in the accuracy or even surpass the
original model, with ~10.5x memory saving and ~3x real inference acceleration.
Both results beat the exiting quantization works with large margins. We extend
our alternating quantization to image classification tasks. In both RNNs and
feedforward neural networks, the method also achieves excellent performance.
