  While recent research endeavors have focused on developing Large Language
Models (LLMs) with robust long-context capabilities, due to the lack of
long-context benchmarks, relatively little is known about how well the
performance of long-context LLMs. To address this gap, we propose a
multi-evidence, position-aware, and scalable benchmark for evaluating
long-context LLMs, named Counting-Stars, which evaluates long-context LLMs by
using two tasks: multi-evidence acquisition and multi-evidence reasoning. Based
on the Counting-Stars test, we conduct experiments to evaluate long-context
LLMs (i.e., GPT-4 Turbo, Gemini 1.5 Pro, Claude3 Opus, GLM-4, and Moonshot-v1).
Experimental results demonstrate that Gemini 1.5 Pro achieves the best overall
results, while the performance of GPT-4 Turbo is the most stable across various
tasks. Furthermore, our analysis of these LLMs, which are extended to handle
long-context scenarios, indicates that there is potential for improvement as
the length of the input context and the intricacy of the tasks are increasing.
