  This paper proposes a novel framework for recurrent neural networks (RNNs)
inspired by the human memory models in the field of cognitive neuroscience to
enhance information processing and transmission between adjacent RNNs' units.
The proposed framework for RNNs consists of three stages that is working
memory, forget, and long-term store. The first stage includes taking input data
into sensory memory and transferring it to working memory for preliminary
treatment. And the second stage mainly focuses on proactively forgetting the
secondary information rather than the primary in the working memory. And
finally, we get the long-term store normally using some kind of RNN's unit. Our
framework, which is generalized and simple, is evaluated on 6 datasets which
fall into 3 different tasks, corresponding to text classification, image
classification and language modelling. Experiments reveal that our framework
can obviously improve the performance of traditional recurrent neural networks.
And exploratory task shows the ability of our framework of correctly forgetting
the secondary information.
