  To advance the evaluation of multimodal math reasoning in large multimodal
models (LMMs), this paper introduces a novel benchmark, MM-MATH. MM-MATH
consists of 5,929 open-ended middle school math problems with visual contexts,
with fine-grained classification across difficulty, grade level, and knowledge
points. Unlike existing benchmarks relying on binary answer comparison, MM-MATH
incorporates both outcome and process evaluations. Process evaluation employs
LMM-as-a-judge to automatically analyze solution steps, identifying and
categorizing errors into specific error types. Extensive evaluation of ten
models on MM-MATH reveals significant challenges for existing LMMs,
highlighting their limited utilization of visual information and struggles with
higher-difficulty problems. The best-performing model achieves only 31%
accuracy on MM-MATH, compared to 82% for humans. This highlights the
challenging nature of our benchmark for existing models and the significant gap
between the multimodal reasoning capabilities of current models and humans. Our
process evaluation reveals that diagram misinterpretation is the most common
error, accounting for more than half of the total error cases, underscoring the
need for improved image comprehension in multimodal reasoning.
