  In this study, we focus on multimodal language understanding for fetching
instructions in the domestic service robots context. This task consists of
predicting a target object, as instructed by the user, given an image and an
unstructured sentence, such as "Bring me the yellow box (from the wooden
cabinet)." This is challenging because of the ambiguity of natural language,
i.e., the relevant information may be missing or there might be several
candidates. To solve such a task, we propose the multimodal target-source
classifier model with attention branches (MTCM-AB), which is an extension of
the MTCM. Our methodology uses the attention branch network (ABN) to develop a
multimodal attention mechanism based on linguistic and visual inputs.
Experimental validation using a standard dataset showed that the MTCM-AB
outperformed both state-of-the-art methods and the MTCM. In particular the
MTCM-AB accuracy on average was 90.1% while human performance was 90.3% on the
PFN-PIC dataset.
