  Understanding expressions that refer to the physical world is crucial for
such human-assisting systems in the real world, as robots that must perform
actions that are expected by users. In real-world reference resolution, a
system must ground the verbal information that appears in user interactions to
the visual information observed in egocentric views. To this end, we propose a
multimodal reference resolution task and construct a Japanese Conversation
dataset for Real-world Reference Resolution (J-CRe3). Our dataset contains
egocentric video and dialogue audio of real-world conversations between two
people acting as a master and an assistant robot at home. The dataset is
annotated with crossmodal tags between phrases in the utterances and the object
bounding boxes in the video frames. These tags include indirect reference
relations, such as predicate-argument structures and bridging references as
well as direct reference relations. We also constructed an experimental model
and clarified the challenges in multimodal reference resolution tasks.
