  Recent advancements in large language models (LLMs) underscore their
potential for responding to inquiries in various domains. However, ensuring
that generative agents provide accurate and reliable answers remains an ongoing
challenge. In this context, multi-agent debate (MAD) has emerged as a promising
strategy for enhancing the truthfulness of LLMs. We benchmark a range of
debating and prompting strategies to explore the trade-offs between cost, time,
and accuracy. Importantly, we find that multi-agent debating systems, in their
current form, do not reliably outperform other proposed prompting strategies,
such as self-consistency and ensembling using multiple reasoning paths.
However, when performing hyperparameter tuning, several MAD systems, such as
Multi-Persona, perform better. This suggests that MAD protocols might not be
inherently worse than other approaches, but that they are more sensitive to
different hyperparameter settings and difficult to optimize. We build on these
results to offer insights into improving debating strategies, such as adjusting
agent agreement levels, which can significantly enhance performance and even
surpass all other non-debate protocols we evaluated. We provide an open-source
repository to the community with several state-of-the-art protocols together
with evaluation scripts to benchmark across popular research datasets.
