  Large pre-trained language models (LMs) have demonstrated impressive
capabilities in generating long, fluent text; however, there is little to no
analysis on their ability to maintain entity coherence and consistency. In this
work, we focus on the end task of narrative generation and systematically
analyse the long-range entity coherence and consistency in generated stories.
First, we propose a set of automatic metrics for measuring model performance in
terms of entity usage. Given these metrics, we quantify the limitations of
current LMs. Next, we propose augmenting a pre-trained LM with a dynamic entity
memory in an end-to-end manner by using an auxiliary entity-related loss for
guiding the reads and writes to the memory. We demonstrate that the dynamic
entity memory increases entity coherence according to both automatic and human
judgment and helps preserving entity-related information especially in settings
with a limited context window. Finally, we also validate that our automatic
metrics are correlated with human ratings and serve as a good indicator of the
quality of generated stories.
