  What would it take for a natural language model to understand a novel, such
as The Lord of the Rings? Among other things, such a model must be able to: (a)
identify and record new characters (entities) and their attributes as they are
introduced in the text, and (b) identify subsequent references to the
characters previously introduced and update their attributes. This problem of
entity tracking is essential for language understanding, and thus, useful for a
wide array of downstream applications in NLP such as question-answering,
summarization.
  In this thesis, we focus on two key problems in relation to facilitating the
use of entity tracking models: (i) scaling entity tracking models to long
documents, such as a novel, and (ii) integrating entity tracking into language
models. Applying language technologies to long documents has garnered interest
recently, but computational constraints are a significant bottleneck in scaling
up current methods. In this thesis, we argue that computationally efficient
entity tracking models can be developed by representing entities with rich,
fixed-dimensional vector representations derived from pretrained language
models, and by exploiting the ephemeral nature of entities. We also argue for
the integration of entity tracking into language models as it will allow for:
(i) wider application given the current ubiquitous use of pretrained language
models in NLP applications, and (ii) easier adoption since it is much easier to
swap in a new pretrained language model than to integrate a separate standalone
entity tracking model.
