  Large Language Models (LLMs) have demonstrated exceptional capabilities in
various natural language tasks, often achieving performances that surpass those
of humans. Despite these advancements, the domain of mathematics presents a
distinctive challenge, primarily due to its specialized structure and the
precision it demands. In this study, we adopted a two-step approach for
investigating the proficiency of LLMs in answering mathematical questions.
First, we employ the most effective LLMs, as identified by their performance on
math question-answer benchmarks, to generate answers to 78 questions from the
Math Stack Exchange (MSE). Second, a case analysis is conducted on the LLM that
showed the highest performance, focusing on the quality and accuracy of its
answers through manual evaluation. We found that GPT-4 performs best (nDCG of
0.48 and P@10 of 0.37) amongst existing LLMs fine-tuned for answering
mathematics questions and outperforms the current best approach on ArqMATH3
Task1, considering P@10. Our Case analysis indicates that while the GPT-4 can
generate relevant responses in certain instances, it does not consistently
answer all questions accurately. This paper explores the current limitations of
LLMs in navigating complex mathematical problem-solving. Through case analysis,
we shed light on the gaps in LLM capabilities within mathematics, thereby
setting the stage for future research and advancements in AI-driven
mathematical reasoning. We make our code and findings publicly available for
research: \url{https://github.com/gipplab/LLM-Investig-MathStackExchange}
