  Transformer-based language models (LMs) are powerful and widely-applicable
tools, but their usefulness is constrained by a finite context window and the
expensive computational cost of processing long text documents. We propose to
adapt pre-trained LMs into AutoCompressors. These language models are capable
of compressing long contexts into compact summary vectors, which are then
accessible to the model as soft prompts. Summary vectors are trained with an
unsupervised objective, whereby long documents are processed in segments, and
summary vectors from all previous segments are used in language modeling. We
fine-tune OPT and Llama-2 models on sequences of up to 30,720 tokens and show
that AutoCompressors can utilize long contexts to improve perplexity. We
evaluate AutoCompressors on in-context learning by compressing task
demonstrations and find that summary vectors are good substitutes for
plain-text demonstrations, increasing accuracy while reducing inference costs.
Finally, we explore the benefits of pre-computing summary vectors for large
corpora by applying summary vectors to retrievalaugmented language modeling and
a passage re-ranking task. Overall, AutoCompressors emerge as a simple and
inexpensive solution to extend the context window of LMs while speeding up
inference over long contexts.
