  We explore a neural network architecture that stacks a recurrent layer and a
feedforward layer that is also connected to the input, and compare it to
standard Elman and LSTM architectures in terms of accuracy and
interpretability. When noise is introduced into the activation function of the
recurrent units, these neurons are forced into a binary activation regime that
makes the networks behave much as finite automata. The resulting models are
simpler, easier to interpret and get higher accuracy on different sample
problems, including the recognition of regular languages, the computation of
additions in different bases and the generation of arithmetic expressions.
