  Neural cache language models (LMs) extend the idea of regular cache language
models by making the cache probability dependent on the similarity between the
current context and the context of the words in the cache. We make an extensive
comparison of 'regular' cache models with neural cache models, both in terms of
perplexity and WER after rescoring first-pass ASR results. Furthermore, we
propose two extensions to this neural cache model that make use of the content
value/information weight of the word: firstly, combining the cache probability
and LM probability with an information-weighted interpolation and secondly,
selectively adding only content words to the cache. We obtain a 29.9%/32.1%
(validation/test set) relative improvement in perplexity with respect to a
baseline LSTM LM on the WikiText-2 dataset, outperforming previous work on
neural cache LMs. Additionally, we observe significant WER reductions with
respect to the baseline model on the WSJ ASR task.
