  Exceptional mathematical reasoning ability is one of the key features that
demonstrate the power of large language models (LLMs). How to comprehensively
define and evaluate the mathematical abilities of LLMs, and even reflect the
user experience in real-world scenarios, has emerged as a critical issue.
Current benchmarks predominantly concentrate on problem-solving capabilities,
which presents a substantial risk of model overfitting and fails to accurately
represent genuine mathematical reasoning abilities. In this paper, we argue
that if a model really understands a problem, it should be robustly and readily
applied across a diverse array of tasks. Motivated by this, we introduce
MATHCHECK, a well-designed checklist for testing task generalization and
reasoning robustness, as well as an automatic tool to generate checklists
efficiently. MATHCHECK includes multiple mathematical reasoning tasks and
robustness test types to facilitate a comprehensive evaluation of both
mathematical reasoning ability and behavior testing. Utilizing MATHCHECK, we
develop MATHCHECK-GSM and MATHCHECK-GEO to assess mathematical textual
reasoning and multi-modal reasoning capabilities, respectively, serving as
upgraded versions of benchmarks including GSM8k, GeoQA, UniGeo, and Geometry3K.
We adopt MATHCHECK-GSM and MATHCHECK-GEO to evaluate over 20 LLMs and 11 MLLMs,
assessing their comprehensive mathematical reasoning abilities. Our results
demonstrate that while frontier LLMs like GPT-4o continue to excel in various
abilities on the checklist, many other model families exhibit a significant
decline. Further experiments indicate that, compared to traditional math
benchmarks, MATHCHECK better reflects true mathematical abilities and
represents mathematical intelligence more linearly, thereby supporting our
design. On our MATHCHECK, we can easily conduct detailed behavior analysis to
deeply investigate models.
