  We introduce a novel and extensible benchmark for large language models
(LLMs) through grid-based games such as Tic-Tac-Toe, Connect Four, and Gomoku.
The open-source game simulation code, available on GitHub, allows LLMs to
compete and generates detailed data files in JSON, CSV, TXT, and PNG formats
for leaderboard rankings and further analysis. We present the results of games
among leading LLMs, including Claude 3.5 Sonnet and Claude 3 Sonnet by
Anthropic, Gemini 1.5 Pro and Gemini 1.5 Flash by Google, GPT-4 Turbo and
GPT-4o by OpenAI, and Llama3-70B by Meta. We also encourage submissions of
results from other LLMs. In total, we simulated 2,310 matches (5 sessions for
each pair among 7 LLMs and a random player) across three types of games, using
three distinct prompt types: list, illustration, and image. The results
revealed significant variations in LLM performance across different games and
prompt types, with analysis covering win and disqualification rates, missed
opportunity analysis, and invalid move analysis. The details of the leaderboard
and result matrix data are available as open-access data on GitHub. This study
enhances our understanding of LLMs' capabilities in playing games they were not
specifically trained for, helping to assess their rule comprehension and
strategic thinking. On the path to Artificial General Intelligence (AGI), this
study lays the groundwork for future exploration into their utility in complex
decision-making scenarios, illuminating their strategic thinking abilities and
offering directions for further inquiry into the limits of LLMs within
game-based frameworks.
