  Although large language models (LLMs) demonstrate impressive performance for
many language tasks, most of them can only handle texts a few thousand tokens
long, limiting their applications on longer sequence inputs, such as books,
reports, and codebases. Recent works have proposed methods to improve LLMs'
long context capabilities by extending context windows and more sophisticated
memory mechanisms. However, comprehensive benchmarks tailored for evaluating
long context understanding are lacking. In this paper, we introduce LongBench,
the first bilingual, multi-task benchmark for long context understanding,
enabling a more rigorous evaluation of long context understanding. LongBench
comprises 21 datasets across 6 task categories in both English and Chinese,
with an average length of 6,711 words (English) and 13,386 characters
(Chinese). These tasks cover key long-text application areas including
single-doc QA, multi-doc QA, summarization, few-shot learning, synthetic tasks,
and code completion. All datasets in LongBench are standardized into a unified
format, allowing for effortless automatic evaluation of LLMs. Upon
comprehensive evaluation of 8 LLMs on LongBench, we find that: (1) Commercial
model (GPT-3.5-Turbo-16k) outperforms other open-sourced models, but still
struggles on longer contexts. (2) Scaled position embedding and fine-tuning on
longer sequences lead to substantial improvement on long context understanding.
(3) Context compression technique such as retrieval brings improvement for
model with weak ability on long contexts, but the performance still lags behind
models that have strong long context understanding capability. The code and
datasets are available at https://github.com/THUDM/LongBench.
