  Large language models have shown impressive results for multi-hop
mathematical reasoning when the input question is only textual. Many
mathematical reasoning problems, however, contain both text and image. With the
ever-increasing adoption of vision language models (VLMs), understanding their
reasoning abilities for such problems is crucial. In this paper, we evaluate
the reasoning capabilities of VLMs along various axes through the lens of
geometry problems. We procedurally create a synthetic dataset of geometry
questions with controllable difficulty levels along multiple axes, thus
enabling a systematic evaluation. The empirical results obtained using our
benchmark for state-of-the-art VLMs indicate that these models are not as
capable in subjects like geometry (and, by generalization, other topics
requiring similar reasoning) as suggested by previous benchmarks. This is made
especially clear by the construction of our benchmark at various depth levels,
since solving higher-depth problems requires long chains of reasoning rather
than additional memorized knowledge. We release the dataset for further
research in this area.
