  Neural language models (LM) trained on diverse corpora are known to work well
on previously seen entities, however, updating these models with dynamically
changing entities such as place names, song titles and shopping items requires
re-training from scratch and collecting full sentences containing these
entities. We aim to address this issue, by introducing entity-aware language
models (EALM), where we integrate entity models trained on catalogues of
entities into the pre-trained LMs. Our combined language model adaptively adds
information from the entity models into the pre-trained LM depending on the
sentence context. Our entity models can be updated independently of the
pre-trained LM, enabling us to influence the distribution of entities output by
the final LM, without any further training of the pre-trained LM. We show
significant perplexity improvements on task-oriented dialogue datasets,
especially on long-tailed utterances, with an ability to continually adapt to
new entities (to an extent).
