  The emergence of Large Language Models (LLMs) has demonstrated promising
progress in solving logical reasoning tasks effectively. Several recent
approaches have proposed to change the role of the LLM from the reasoner into a
translator between natural language statements and symbolic representations
which are then sent to external symbolic solvers to resolve. This paradigm has
established the current state-of-the-art result in logical reasoning (i.e.,
deductive reasoning). However, it remains unclear whether the variance in
performance of these approaches stems from the methodologies employed or the
specific symbolic solvers utilized. There is a lack of consistent comparison
between symbolic solvers and how they influence the overall reported
performance. This is important, as each symbolic solver also has its own input
symbolic language, presenting varying degrees of challenge in the translation
process. To address this gap, we perform experiments on 3 deductive reasoning
benchmarks with LLMs augmented with widely used symbolic solvers: Z3, Pyke, and
Prover9. The tool-executable rates of symbolic translation generated by
different LLMs exhibit a near 50% performance variation. This highlights a
significant difference in performance rooted in very basic choices of tools.
The almost linear correlation between the executable rate of translations and
the accuracy of the outcomes from Prover9 highlight a strong alignment between
LLMs ability to translate into Prover9 symbolic language, and the correctness
of those translations.
