  AI researchers have posited Dungeons and Dragons (D&D) as a challenge problem
to test systems on various language-related capabilities. In this paper, we
frame D&D specifically as a dialogue system challenge, where the tasks are to
both generate the next conversational turn in the game and predict the state of
the game given the dialogue history. We create a gameplay dataset consisting of
nearly 900 games, with a total of 7,000 players, 800,000 dialogue turns,
500,000 dice rolls, and 58 million words. We automatically annotate the data
with partial state information about the game play. We train a large language
model (LM) to generate the next game turn, conditioning it on different
information. The LM can respond as a particular character or as the player who
runs the game--i.e., the Dungeon Master (DM). It is trained to produce dialogue
that is either in-character (roleplaying in the fictional world) or
out-of-character (discussing rules or strategy). We perform a human evaluation
to determine what factors make the generated output plausible and interesting.
We further perform an automatic evaluation to determine how well the model can
predict the game state given the history and examine how well tracking the game
state improves its ability to produce plausible conversational output.
