  Much of NLP research has focused on crowdsourced static datasets and the
supervised learning paradigm of training once and then evaluating test
performance. As argued in de Vries et al. (2020), crowdsourced data has the
issues of lack of naturalness and relevance to real-world use cases, while the
static dataset paradigm does not allow for a model to learn from its
experiences of using language (Silver et al., 2013). In contrast, one might
hope for machine learning systems that become more useful as they interact with
people. In this work, we build and deploy a role-playing game, whereby human
players converse with learning agents situated in an open-domain fantasy world.
We show that by training models on the conversations they have with humans in
the game the models progressively improve, as measured by automatic metrics and
online engagement scores. This learning is shown to be more efficient than
crowdsourced data when applied to conversations with real users, as well as
being far cheaper to collect.
