  Recent advancements in Large Language Models (LLMs) showcase advanced
reasoning, yet NLP evaluations often depend on static benchmarks. Evaluating
this necessitates environments that test strategic reasoning in dynamic,
competitive scenarios requiring long-term planning. We introduce AucArena, a
novel evaluation suite that simulates auctions, a setting chosen for being
highly unpredictable and involving many skills related to resource and risk
management, while also being easy to evaluate. We conduct controlled
experiments using state-of-the-art LLMs to power bidding agents to benchmark
their planning and execution skills. Our research demonstrates that LLMs, such
as GPT-4, possess key skills for auction participation, such as budget
management and goal adherence, which improve with adaptive strategies. This
highlights LLMs' potential in modeling complex social interactions in
competitive contexts. However, variability in LLM performance and occasional
outperformance by simpler methods indicate opportunities for further
advancements in LLM design and the value of our simulation environment for
ongoing testing and refinement.
