  Dialogue research tends to distinguish between chit-chat and goal-oriented
tasks. While the former is arguably more naturalistic and has a wider use of
language, the latter has clearer metrics and a straightforward learning signal.
Humans effortlessly combine the two, for example engaging in chit-chat with the
goal of exchanging information or eliciting a specific response. Here, we
bridge the divide between these two domains in the setting of a rich
multi-player text-based fantasy environment where agents and humans engage in
both actions and dialogue. Specifically, we train a goal-oriented model with
reinforcement learning against an imitation-learned ``chit-chat'' model with
two approaches: the policy either learns to pick a topic or learns to pick an
utterance given the top-K utterances from the chit-chat model. We show that
both models outperform an inverse model baseline and can converse naturally
with their dialogue partner in order to achieve goals.
