  Developing Large Language Models (LLMs) with robust long-context capabilities
has been the recent research focus, resulting in the emergence of long-context
LLMs proficient in Chinese. However, the evaluation of these models remains
underdeveloped due to a lack of benchmarks. To address this gap, we present
CLongEval, a comprehensive Chinese benchmark for evaluating long-context LLMs.
CLongEval is characterized by three key features: (1) Sufficient data volume,
comprising 7 distinct tasks and 7,267 examples; (2) Broad applicability,
accommodating to models with context windows size from 1K to 100K; (3) High
quality, with over 2,000 manually annotated question-answer pairs in addition
to the automatically constructed labels. With CLongEval, we undertake a
comprehensive assessment of 6 open-source long-context LLMs and 2 leading
commercial counterparts that feature both long-context abilities and
proficiency in Chinese. We also provide in-depth analysis based on the
empirical results, trying to shed light on the critical capabilities that
present challenges in long-context settings. The dataset, evaluation scripts,
and model outputs will be released.
