  The use of generative AI in video game development is on the rise, and as the
conversational and other capabilities of large language models continue to
improve, we expect LLM-driven non-player characters (NPCs) to become widely
deployed. In this paper, we seek to understand how human players collaborate
with LLM-driven NPCs to accomplish in-game goals. We design a minigame within
Minecraft where a player works with two GPT4-driven NPCs to complete a quest.
We perform a user study in which 28 Minecraft players play this minigame and
share their feedback. On analyzing the game logs and recordings, we find that
several patterns of collaborative behavior emerge from the NPCs and the human
players. We also report on the current limitations of language-only models that
do not have rich game-state or visual understanding. We believe that this
preliminary study and analysis will inform future game developers on how to
better exploit these rapidly improving generative AI models for collaborative
roles in games.
