  Natural language understanding often requires deep semantic knowledge.
Expanding on previous proposals, we suggest that some important aspects of
semantic knowledge can be modeled as a language model if done at an appropriate
level of abstraction. We develop two distinct models that capture semantic
frame chains and discourse information while abstracting over the specific
mentions of predicates and entities. For each model, we investigate four
implementations: a "standard" N-gram language model and three discriminatively
trained "neural" language models that generate embeddings for semantic frames.
The quality of the semantic language models (SemLM) is evaluated both
intrinsically, using perplexity and a narrative cloze test and extrinsically -
we show that our SemLM helps improve performance on semantic natural language
processing tasks such as co-reference resolution and discourse parsing.
