  Recurrent neural networks are a widely used class of neural architectures.
They have, however, two shortcomings. First, they are often treated as
black-box models and as such it is difficult to understand what exactly they
learn as well as how they arrive at a particular prediction. Second, they tend
to work poorly on sequences requiring long-term memorization, despite having
this capacity in principle. We aim to address both shortcomings with a class of
recurrent networks that use a stochastic state transition mechanism between
cell applications. This mechanism, which we term state-regularization, makes
RNNs transition between a finite set of learnable states. We evaluate
state-regularized RNNs on (1) regular languages for the purpose of automata
extraction; (2) non-regular languages such as balanced parentheses and
palindromes where external memory is required; and (3) real-word sequence
learning tasks for sentiment analysis, visual object recognition and text
categorisation. We show that state-regularization (a) simplifies the extraction
of finite state automata that display an RNN's state transition dynamic; (b)
forces RNNs to operate more like automata with external memory and less like
finite state machines, which potentiality leads to a more structural memory;
(c) leads to better interpretability and explainability of RNNs by leveraging
the probabilistic finite state transition mechanism over time steps.
