  We study the understanding of embodied reference: One agent uses both
language and gesture to refer to an object to another agent in a shared
physical environment. Of note, this new visual task requires understanding
multimodal cues with perspective-taking to identify which object is being
referred to. To tackle this problem, we introduce YouRefIt, a new crowd-sourced
dataset of embodied reference collected in various physical scenes; the dataset
contains 4,195 unique reference clips in 432 indoor scenes. To the best of our
knowledge, this is the first embodied reference dataset that allows us to study
referring expressions in daily physical scenes to understand referential
behavior, human communication, and human-robot interaction. We further devise
two benchmarks for image-based and video-based embodied reference
understanding. Comprehensive baselines and extensive experiments provide the
very first result of machine perception on how the referring expressions and
gestures affect the embodied reference understanding. Our results provide
essential evidence that gestural cues are as critical as language cues in
understanding the embodied reference.
