  Logical reasoning has been an ongoing pursuit in the field of AI. Despite
significant advancements made by large language models (LLMs), they still
struggle with complex logical reasoning problems. To enhance reasoning
performance, one promising direction is scalable oversight, which requires LLMs
to identify their own errors and then improve by themselves. Various
self-verification methods have been proposed in pursuit of this goal.
Nevertheless, whether existing models understand their own errors well is still
under investigation. In this paper, we take a closer look at the
self-verification abilities of LLMs in the context of logical reasoning,
focusing on their ability to identify logical fallacies accurately. We
introduce a dataset, FALLACIES, containing 232 types of reasoning fallacies
categorized in a hierarchical taxonomy. By conducting exhaustive experiments on
FALLACIES, we obtain comprehensive and detailed analyses of a series of models
on their verification abilities. Our main findings suggest that existing LLMs
could struggle to identify fallacious reasoning steps accurately and may fall
short of guaranteeing the validity of self-verification methods. Drawing from
these observations, we offer suggestions for future research and practical
applications of self-verification methods.
