  A common technique for ameliorating the computational costs of running large
neural models is sparsification, or the removal of neural connections during
training. Sparse models are capable of maintaining the high accuracy of state
of the art models, while functioning at the cost of more parsimonious models.
The structures which underlie sparse architectures are, however, poorly
understood and not consistent between differently trained models and
sparsification schemes. In this paper, we propose a new technique for
sparsification of recurrent neural nets (RNNs), called moduli regularization,
in combination with magnitude pruning. Moduli regularization leverages the
dynamical system induced by the recurrent structure to induce a geometric
relationship between neurons in the hidden state of the RNN. By making our
regularizing term explicitly geometric, we provide the first, to our knowledge,
a priori description of the desired sparse architecture of our neural net. We
verify the effectiveness of our scheme for navigation and natural language
processing RNNs. Navigation is a structurally geometric task, for which there
are known moduli spaces, and we show that regularization can be used to reach
90% sparsity while maintaining model performance only when coefficients are
chosen in accordance with a suitable moduli space. Natural language processing,
however, has no known moduli space in which computations are performed.
Nevertheless, we show that moduli regularization induces more stable recurrent
neural nets with a variety of moduli regularizers, and achieves high fidelity
models at 98% sparsity.
