  Where early work on dialogue in Computational Linguistics put much emphasis
on dialogue structure and its relation to the mental states of the dialogue
participants (e.g., Allen 1979, Grosz & Sidner 1986), current work mostly
reduces dialogue to the task of producing at any one time a next utterance;
e.g. in neural chatbot or Visual Dialogue settings. As a methodological
decision, this is sound: Even the longest journey is a sequence of steps. It
becomes detrimental, however, when the tasks and datasets from which dialogue
behaviour is to be learned are tailored too much to this framing of the
problem. In this short note, we describe a family of settings which still allow
to keep dialogues simple, but add a constraint that makes participants care
about reaching mutual understanding. In such agreement games, there is a
secondary, but explicit goal besides the task level goal, and that is to reach
mutual understanding about whether the task level goal has been reached. As we
argue, this naturally triggers meta-semantic interaction and mutual engagement,
and hence leads to richer data from which to induce models.
