  It has been established in recent work that Large Language Models (LLMs) can
be prompted to "self-play" conversational games that probe certain capabilities
(general instruction following, strategic goal orientation, language
understanding abilities), where the resulting interactive game play can be
automatically scored. In this paper, we take one of the proposed frameworks for
setting up such game-play environments, and further test its usefulness as an
evaluation instrument, along a number of dimensions: We show that it can easily
keep up with new developments while avoiding data contamination, we show that
the tests implemented within it are not yet saturated (human performance is
substantially higher than that of even the best models), and we show that it
lends itself to investigating additional questions, such as the impact of the
prompting language on performance. We believe that the approach forms a good
basis for making decisions on model choice for building applied interactive
systems, and perhaps ultimately setting up a closed-loop development
environment of system and simulated evaluator.
