  We present LoCoVQA, a dynamic benchmark generator for evaluating long-context
extractive reasoning in vision language models (VLMs). LoCoVQA augments test
examples for mathematical reasoning, VQA, and character recognition tasks with
increasingly long visual contexts composed of both in-distribution and
out-of-distribution distractor images.
  Across these tasks, a diverse set of VLMs rapidly lose performance as the
visual context length grows, often exhibiting a striking logarithmic decay
trend. This test assesses how well VLMs can ignore irrelevant information when
answering queries -- a task that is quite easy for language models (LMs) in the
text domain -- demonstrating that current state-of-the-art VLMs lack this
essential capability for many long-context applications.
