  Logical reasoning consistently plays a fundamental and significant role in
the domains of knowledge engineering and artificial intelligence. Recently,
Large Language Models (LLMs) have emerged as a noteworthy innovation in natural
language processing (NLP), exhibiting impressive achievements across various
classic NLP tasks. However, the question of whether LLMs can effectively
address the task of logical reasoning, which requires gradual cognitive
inference similar to human intelligence, remains unanswered. To this end, we
aim to bridge this gap and provide comprehensive evaluations in this paper.
Firstly, to offer systematic evaluations, we select fifteen typical logical
reasoning datasets and organize them into deductive, inductive, abductive and
mixed-form reasoning settings. Considering the comprehensiveness of
evaluations, we include three representative LLMs (i.e., text-davinci-003,
ChatGPT and BARD) and evaluate them on all selected datasets under zero-shot,
one-shot and three-shot settings. Secondly, different from previous evaluations
relying only on simple metrics (e.g., accuracy), we propose fine-level
evaluations from objective and subjective manners, covering both answers and
explanations. Additionally, to uncover the logical flaws of LLMs, problematic
cases will be attributed to five error types from two dimensions, i.e.,
evidence selection process and reasoning process. Thirdly, to avoid the
influences of knowledge bias and purely focus on benchmarking the logical
reasoning capability of LLMs, we propose a new dataset with neutral content. It
contains 3,000 samples and covers deductive, inductive and abductive settings.
Based on the in-depth evaluations, this paper finally forms a general
evaluation scheme of logical reasoning capability from six dimensions. It
reflects the pros and cons of LLMs and gives guiding directions for future
works.
