  Video understanding is a crucial next step for multimodal large language
models (MLLMs). To probe specific aspects of video understanding ability,
existing video benchmarks typically require careful video selection based on
the target capability, along with laborious annotation of query-response pairs
to match the specific video content. This process is both challenging and
resource-intensive. In this paper, we propose VideoNIAH (Video Needle In A
Haystack), a benchmark construction framework through synthetic video
generation. VideoNIAH decouples test video content from their query-responses
by inserting unrelated image/text 'needles' into original videos. It generates
annotations solely from these needles, ensuring diversity in video sources and
a variety of query-responses. Additionally, by inserting multiple needles,
VideoNIAH rigorously evaluates the temporal understanding capabilities of
models. We utilized VideoNIAH to compile a video benchmark VNBench, including
tasks such as retrieval, ordering, and counting. VNBench can efficiently
evaluate the fine-grained understanding ability and spatio-temporal modeling
ability of a video model, while also supporting the long-context evaluation.
Additionally, we evaluated recent video-centric multimodal large language
models (MLLMs), both open-source and proprietary, providing a comprehensive
analysis. We found that although proprietary models have significant advantages
over open-source models, all existing video models still perform poorly on
long-distance dependency tasks. VideoNIAH is a simple yet highly scalable
benchmark construction framework, and we believe it will inspire future video
benchmark works. The code and data are available at
https://github.com/joez17/VideoNIAH.
