  Dialogue agents that interact with humans in situated environments need to
manage referential ambiguity across multiple modalities and ask for help as
needed. However, it is not clear what kinds of questions such agents should ask
nor how the answers to such questions can be used to resolve ambiguity. To
address this, we analyzed dialogue data from an interactive study in which
participants controlled a virtual robot tasked with organizing a set of tools
while engaging in dialogue with a live, remote experimenter. We discovered a
number of novel results, including the distribution of question types used to
resolve ambiguity and the influence of dialogue-level factors on the reference
resolution process. Based on these empirical findings we: (1) developed a
computational model for clarification requests using a decision network with an
entropy-based utility assignment method that operates across modalities, (2)
evaluated the model, showing that it outperforms a slot-filling baseline in
environments of varying ambiguity, and (3) interpreted the results to offer
insight into the ways that agents can ask questions to facilitate situated
reference resolution.
