  We present a new recurrent neural network topology to enhance
state-of-the-art machine learning systems by incorporating a broader context.
Our approach overcomes recent limitations with extended narratives through a
multi-layered computational approach to generate an abstract context
representation. Therefore, the developed system captures the narrative on
word-level, sentence-level, and context-level. Through the hierarchical set-up,
our proposed model summarizes the most salient information on each level and
creates an abstract representation of the extended context. We subsequently use
this representation to enhance neural language processing systems on the task
of semantic error detection. To show the potential of the newly introduced
topology, we compare the approach against a context-agnostic set-up including a
standard neural language model and a supervised binary classification network.
The performance measures on the error detection task show the advantage of the
hierarchical context-aware topologies, improving the baseline by 12.75%
relative for unsupervised models and 20.37% relative for supervised models.
