  Model compression is significant for the wide adoption of Recurrent Neural
Networks (RNNs) in both user devices possessing limited resources and business
clusters requiring quick responses to large-scale service requests. This work
aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the
sizes of basic structures within LSTM units, including input updates, gates,
hidden states, cell states and outputs. Independently reducing the sizes of
basic structures can result in inconsistent dimensions among them, and
consequently, end up with invalid LSTM units. To overcome the problem, we
propose Intrinsic Sparse Structures (ISS) in LSTMs. Removing a component of ISS
will simultaneously decrease the sizes of all basic structures by one and
thereby always maintain the dimension consistency. By learning ISS within LSTM
units, the obtained LSTMs remain regular while having much smaller basic
structures. Based on group Lasso regularization, our method achieves 10.59x
speedup without losing any perplexity of a language modeling of Penn TreeBank
dataset. It is also successfully evaluated through a compact model with only
2.69M weights for machine Question Answering of SQuAD dataset. Our approach is
successfully extended to non- LSTM RNNs, like Recurrent Highway Networks
(RHNs). Our source code is publicly available at
https://github.com/wenwei202/iss-rnns
