  Although self-attention based models such as Transformers have achieved
remarkable successes on natural language processing (NLP) tasks, recent studies
reveal that they have limitations on modeling sequential transformations (Hahn,
2020), which may prompt re-examinations of recurrent neural networks (RNNs)
that demonstrated impressive results on handling sequential data. Despite many
prior attempts to interpret RNNs, their internal mechanisms have not been fully
understood, and the question on how exactly they capture sequential features
remains largely unclear. In this work, we present a study that shows there
actually exist some explainable components that reside within the hidden
states, which are reminiscent of the classical n-grams features. We evaluated
such extracted explainable features from trained RNNs on downstream sentiment
analysis tasks and found they could be used to model interesting linguistic
phenomena such as negation and intensification. Furthermore, we examined the
efficacy of using such n-gram components alone as encoders on tasks such as
sentiment analysis and language modeling, revealing they could be playing
important roles in contributing to the overall performance of RNNs. We hope our
findings could add interpretability to RNN architectures, and also provide
inspirations for proposing new architectures for sequential data.
