  Large language models (LLMs) demonstrate substantial capabilities in solving
math problems. However, they tend to produce hallucinations when given
questions containing unreasonable errors. In this paper, we study the behavior
of LLMs when faced with unreasonable math problems and further explore their
potential to address these problems. We construct the Unreasonable Math Problem
(UMP) benchmark to examine the error detection ability of LLMs. Experiments
show that LLMs are able to detect unreasonable errors, but still fail in
generating non-hallucinatory content. In order to improve their ability of
error detection and correction, we further design a strategic prompt template
called Critical Calculation and Conclusion(CCC). With CCC, LLMs can better
self-evaluate and detect unreasonable errors in math questions, making them
more reliable and safe in practical application scenarios.
