  Neural networks are becoming a popular tool for solving many real-world
problems such as object recognition and machine translation, thanks to its
exceptional performance as an end-to-end solution. However, neural networks are
complex black-box models, which hinders humans from interpreting and
consequently trusting them in making critical decisions. Towards interpreting
neural networks, several approaches have been proposed to extract simple
deterministic models from neural networks. The results are not encouraging
(e.g., low accuracy and limited scalability), fundamentally due to the limited
expressiveness of such simple models.
  In this work, we propose an approach to extract probabilistic automata for
interpreting an important class of neural networks, i.e., recurrent neural
networks. Our work distinguishes itself from existing approaches in two
important ways. One is that probability is used to compensate for the loss of
expressiveness. This is inspired by the observation that human reasoning is
often `probabilistic'. The other is that we adaptively identify the right level
of abstraction so that a simple model is extracted in a request-specific way.
We conduct experiments on several real-world datasets using state-of-the-art
architectures including GRU and LSTM. The result shows that our approach
significantly improves existing approaches in terms of accuracy or scalability.
Lastly, we demonstrate the usefulness of the extracted models through detecting
adversarial texts.
