  Dungeons & Dragons (D&D) is a tabletop roleplaying game with complex natural
language interactions between players and hidden state information. Recent work
has shown that large language models (LLMs) that have access to state
information can generate higher quality game turns than LLMs that use dialog
history alone. However, previous work used game state information that was
heuristically created and was not a true gold standard game state. We present
FIREBALL, a large dataset containing nearly 25,000 unique sessions from real
D&D gameplay on Discord with true game state info. We recorded game play
sessions of players who used the Avrae bot, which was developed to aid people
in playing D&D online, capturing language, game commands and underlying game
state information. We demonstrate that FIREBALL can improve natural language
generation (NLG) by using Avrae state information, improving both automated
metrics and human judgments of quality. Additionally, we show that LLMs can
generate executable Avrae commands, particularly after finetuning.
