  The behavior of Large Language Models (LLMs) as artificial social agents is
largely unexplored, and we still lack extensive evidence of how these agents
react to simple social stimuli. Testing the behavior of AI agents in classic
Game Theory experiments provides a promising theoretical framework for
evaluating the norms and values of these agents in archetypal social
situations. In this work, we investigate the cooperative behavior of Llama2
when playing the Iterated Prisoner's Dilemma against random adversaries
displaying various levels of hostility. We introduce a systematic methodology
to evaluate an LLM's comprehension of the game's rules and its capability to
parse historical gameplay logs for decision-making. We conducted simulations of
games lasting for 100 rounds, and analyzed the LLM's decisions in terms of
dimensions defined in behavioral economics literature. We find that Llama2
tends not to initiate defection but it adopts a cautious approach towards
cooperation, sharply shifting towards a behavior that is both forgiving and
non-retaliatory only when the opponent reduces its rate of defection below 30%.
In comparison to prior research on human participants, Llama2 exhibits a
greater inclination towards cooperative behavior. Our systematic approach to
the study of LLMs in game theoretical scenarios is a step towards using these
simulations to inform practices of LLM auditing and alignment.
