  Large language models (LLMs) have significantly advanced natural language
understanding and demonstrated strong problem-solving abilities. Despite these
successes, most LLMs still struggle with solving mathematical problems due to
the intricate reasoning required. This paper investigates the mathematical
problem-solving capabilities of LLMs using the newly developed "MathOdyssey"
dataset. The dataset includes diverse mathematical problems at high school and
university levels, created by experts from notable institutions to rigorously
test LLMs in advanced problem-solving scenarios and cover a wider range of
subject areas. By providing the MathOdyssey dataset as a resource to the AI
community, we aim to contribute to the understanding and improvement of AI
capabilities in complex mathematical problem-solving. We conduct benchmarking
on open-source models, such as Llama-3 and DBRX-Instruct, and closed-source
models from the GPT series and Gemini models. Our results indicate that while
LLMs perform well on routine and moderately difficult tasks, they face
significant challenges with Olympiad-level problems and complex
university-level questions. Our analysis shows a narrowing performance gap
between open-source and closed-source models, yet substantial challenges
remain, particularly with the most demanding problems. This study highlights
the ongoing need for research to enhance the mathematical reasoning of LLMs.
The dataset, results, and code are publicly available.
