  Recurrent neural networks (RNNs) have been widely adopted in temporal
sequence analysis, where realtime performance is often in demand. However, RNNs
suffer from heavy computational workload as the model often comes with large
weight matrices. Pruning schemes have been proposed for RNNs to eliminate the
redundant (close-to-zero) weight values. On one hand, the non-structured
pruning methods achieve a high pruning rate but introducing computation
irregularity (random sparsity), which is unfriendly to parallel hardware. On
the other hand, hardware-oriented structured pruning suffers from low pruning
rate due to restricted constraints on allowable pruning structure. This paper
presents CSB-RNN, an optimized full-stack RNN framework with a novel compressed
structured block (CSB) pruning technique. The CSB pruned RNN model comes with
both fine pruning granularity that facilitates a high pruning rate and regular
structure that benefits the hardware parallelism. To address the challenges in
parallelizing the CSB pruned model inference with fine-grained structural
sparsity, we propose a novel hardware architecture with a dedicated compiler.
Gaining from the architecture-compilation co-design, the hardware not only
supports various RNN cell types, but is also able to address the challenging
workload imbalance issue and therefore significantly improves the hardware
efficiency.
