  The interpretability of deep learning models has raised extended attention
these years. It will be beneficial if we can learn an interpretable structure
from deep learning models. In this paper, we focus on Recurrent Neural
Networks~(RNNs) especially gated RNNs whose inner mechanism is still not
clearly understood. We find that Finite State Automaton~(FSA) that processes
sequential data has more interpretable inner mechanism according to the
definition of interpretability and can be learned from RNNs as the
interpretable structure. We propose two methods to learn FSA from RNN based on
two different clustering methods. With the learned FSA and via experiments on
artificial and real datasets, we find that FSA is more trustable than the RNN
from which it learned, which gives FSA a chance to substitute RNNs in
applications involving humans' lives or dangerous facilities. Besides, we
analyze how the number of gates affects the performance of RNN. Our result
suggests that gate in RNN is important but the less the better, which could be
a guidance to design other RNNs. Finally, we observe that the FSA learned from
RNN gives semantic aggregated states and its transition graph shows us a very
interesting vision of how RNNs intrinsically handle text classification tasks.
