  Recently, continuous cache models were proposed as extensions to recurrent
neural network language models, to adapt their predictions to local changes in
the data distribution. These models only capture the local context, of up to a
few thousands tokens. In this paper, we propose an extension of continuous
cache models, which can scale to larger contexts. In particular, we use a large
scale non-parametric memory component that stores all the hidden activations
seen in the past. We leverage recent advances in approximate nearest neighbor
search and quantization algorithms to store millions of representations while
searching them efficiently. We conduct extensive experiments showing that our
approach significantly improves the perplexity of pre-trained language models
on new distributions, and can scale efficiently to much larger contexts than
previously proposed local cache models.
