  In this work, we propose a novel method to incorporate corpus-level discourse
information into language modelling. We call this larger-context language
model. We introduce a late fusion approach to a recurrent language model based
on long short-term memory units (LSTM), which helps the LSTM unit keep
intra-sentence dependencies and inter-sentence dependencies separate from each
other. Through the evaluation on three corpora (IMDB, BBC, and PennTree Bank),
we demon- strate that the proposed model improves perplexity significantly. In
the experi- ments, we evaluate the proposed approach while varying the number
of context sentences and observe that the proposed late fusion is superior to
the usual way of incorporating additional inputs to the LSTM. By analyzing the
trained larger- context language model, we discover that content words,
including nouns, adjec- tives and verbs, benefit most from an increasing number
of context sentences. This analysis suggests that larger-context language model
improves the unconditional language model by capturing the theme of a document
better and more easily.
