  Video game testing requires game-specific knowledge as well as common sense
reasoning about the events in the game. While AI-driven agents can satisfy the
first requirement, it is not yet possible to meet the second requirement
automatically. Therefore, video game testing often still relies on manual
testing, and human testers are required to play the game thoroughly to detect
bugs. As a result, it is challenging to fully automate game testing. In this
study, we explore the possibility of leveraging the zero-shot capabilities of
large language models for video game bug detection. By formulating the bug
detection problem as a question-answering task, we show that large language
models can identify which event is buggy in a sequence of textual descriptions
of events from a game. To this end, we introduce the GameBugDescriptions
benchmark dataset, which consists of 167 buggy gameplay videos and a total of
334 question-answer pairs across 8 games. We extensively evaluate the
performance of six models across the OPT and InstructGPT large language model
families on our benchmark dataset. Our results show promising results for
employing language models to detect video game bugs. With the proper prompting
technique, we could achieve an accuracy of 70.66%, and on some video games, up
to 78.94%. Our code, evaluation data and the benchmark can be found on
https://asgaardlab.github.io/LLMxBugs
