  The ability to understand and reason about spatial relationships between
objects in images is an important component of visual reasoning. This skill
rests on the ability to recognize and localize objects of interest and
determine their spatial relation. Early vision and language models (VLMs) have
been shown to struggle to recognize spatial relations. We extend the previously
released What'sUp dataset and propose a novel comprehensive evaluation for
spatial relationship understanding that highlights the strengths and weaknesses
of 27 different models. In addition to the VLMs evaluated in What'sUp, our
extensive evaluation encompasses 3 classes of Multimodal LLMs (MLLMs) that vary
in their parameter sizes (ranging from 7B to 110B), training/instruction-tuning
methods, and visual resolution to benchmark their performances and scrutinize
the scaling laws in this task.
