  In this paper, we explore the potential of Large Language Models (LLMs)
Agents in playing the strategic social deduction game, Resistance Avalon.
Players in Avalon are challenged not only to make informed decisions based on
dynamically evolving game phases, but also to engage in discussions where they
must deceive, deduce, and negotiate with other players. These characteristics
make Avalon a compelling test-bed to study the decision-making and
language-processing capabilities of LLM Agents. To facilitate research in this
line, we introduce AvalonBench - a comprehensive game environment tailored for
evaluating multi-agent LLM Agents. This benchmark incorporates: (1) a game
environment for Avalon, (2) rule-based bots as baseline opponents, and (3)
ReAct-style LLM agents with tailored prompts for each role. Notably, our
evaluations based on AvalonBench highlight a clear capability gap. For
instance, models like ChatGPT playing good-role got a win rate of 22.2% against
rule-based bots playing evil, while good-role bot achieves 38.2% win rate in
the same setting. We envision AvalonBench could be a good test-bed for
developing more advanced LLMs (with self-playing) and agent frameworks that can
effectively model the layered complexities of such game environments.
