  It is essential for dialogue-based spatial reasoning systems to maintain
memory of historical states of the world. In addition to conveying that the
dialogue agent is mentally present and engaged with the task, referring to
historical states may be crucial for enabling collaborative planning (e.g., for
planning to return to a previous state, or diagnosing a past misstep). In this
paper, we approach the problem of spatial memory in a multi-modal spoken
dialogue system capable of answering questions about interaction history in a
physical blocks world setting. This work builds upon a full spatial
question-answering pipeline consisting of a vision system, speech input and
output mediated by an animated avatar, a dialogue system that robustly
interprets spatial queries, and a constraint solver that derives answers based
on 3-D spatial modelling. The contributions of this work include a symbolic
dialogue context registering knowledge about discourse history and changes in
the world, as well as a natural language understanding module capable of
interpreting free-form historical questions and querying the dialogue context
to form an answer.
