  Large language models (LLMs) are highly effective in various natural language
processing (NLP) tasks. However, they are susceptible to producing unreliable
conjectures in ambiguous contexts called hallucination. This paper presents a
new method for evaluating LLM hallucination in Question Answering (QA) based on
the unanswerable math word problem (MWP). To support this approach, we
innovatively develop a dataset called Unanswerable Math Word Problem (UMWP)
which comprises 5200 questions across five categories. We developed an
evaluation methodology combining text similarity and mathematical expression
detection to determine whether LLM considers the question unanswerable. The
results of extensive experiments conducted on 31 LLMs, including GPT-3,
InstructGPT, LLaMA, and Claude, demonstrate that in-context learning and
reinforcement learning with human feedback (RLHF) training significantly
enhance the model's ability to avoid hallucination. We show that utilizing MWP
is a reliable and effective approach to assess hallucination. Our code and data
are available at https://github.com/Yuki-Asuuna/UMWP.
