  Recurrent neural networks (RNNs) are temporal networks and cumulative in
nature that have shown promising results in various natural language processing
tasks. Despite their success, it still remains a challenge to understand their
hidden behavior. In this work, we analyze and interpret the cumulative nature
of RNN via a proposed technique named as Layer-wIse-Semantic-Accumulation
(LISA) for explaining decisions and detecting the most likely (i.e., saliency)
patterns that the network relies on while decision making. We demonstrate (1)
LISA: "How an RNN accumulates or builds semantics during its sequential
processing for a given text example and expected response" (2) Example2pattern:
"How the saliency patterns look like for each category in the data according to
the network in decision making". We analyse the sensitiveness of RNNs about
different inputs to check the increase or decrease in prediction scores and
further extract the saliency patterns learned by the network. We employ two
relation classification datasets: SemEval 10 Task 8 and TAC KBP Slot Filling to
explain RNN predictions via the LISA and example2pattern.
