  Recurrent neural networks (RNNs) have shown excellent performance in
processing sequence data. However, they are both complex and memory intensive
due to their recursive nature. These limitations make RNNs difficult to embed
on mobile devices requiring real-time processes with limited hardware
resources. To address the above issues, we introduce a method that can learn
binary and ternary weights during the training phase to facilitate hardware
implementations of RNNs. As a result, using this approach replaces all
multiply-accumulate operations by simple accumulations, bringing significant
benefits to custom hardware in terms of silicon area and power consumption. On
the software side, we evaluate the performance (in terms of accuracy) of our
method using long short-term memories (LSTMs) on various sequential models
including sequence classification and language modeling. We demonstrate that
our method achieves competitive results on the aforementioned tasks while using
binary/ternary weights during the runtime. On the hardware side, we present
custom hardware for accelerating the recurrent computations of LSTMs with
binary/ternary weights. Ultimately, we show that LSTMs with binary/ternary
weights can achieve up to 12x memory saving and 10x inference speedup compared
to the full-precision implementation on an ASIC platform.
