  We present the Compressive Transformer, an attentive sequence model which
compresses past memories for long-range sequence learning. We find the
Compressive Transformer obtains state-of-the-art language modelling results in
the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc
respectively. We also find it can model high-frequency speech effectively and
can be used as a memory mechanism for RL, demonstrated on an object matching
task. To promote the domain of long-range sequence learning, we propose a new
open-vocabulary language modelling benchmark derived from books, PG-19.
