  Large language models (LLMs) such as ChatGPT and GPT-4 have shown impressive
performance in complex reasoning tasks. However, it is difficult to know
whether the models are reasoning based on deep understandings of truth and
logic, or leveraging their memorized patterns in a relatively superficial way.
In this work, we explore testing LLMs' reasoning by engaging with them in a
debate-like conversation, where given a question, the LLM and the user need to
discuss to make the correct decision starting from opposing arguments. Upon
mitigating the Clever Hans effect, our task requires the LLM to not only
achieve the correct answer on its own, but also be able to hold and defend its
belief instead of blindly believing or getting misled by the user's (invalid)
arguments and critiques, thus testing in greater depth whether the LLM grasps
the essence of the reasoning required to solve the problem. Across a range of
complex reasoning benchmarks spanning math, commonsense, logic and BIG-Bench
tasks, we find that despite their impressive performance as reported in
existing work on generating correct step-by-step solutions in the beginning,
LLMs like ChatGPT cannot maintain their beliefs in truth for a significant
portion of examples when challenged by oftentimes absurdly invalid arguments.
Our work points to danger zones of model alignment, and also suggests more
careful treatments and interpretations of the recent findings that LLMs can
improve their responses based on feedback.
