  This paper aims to develop a framework that enables a robot to execute tasks
based on visual information, in response to natural language instructions for
Fetch-and-Carry with Object Grounding (FCOG) tasks. Although there have been
many frameworks, they usually rely on manually given instruction sentences.
Therefore, evaluations have only been conducted with fixed tasks. Furthermore,
many multimodal language understanding models for the benchmarks only consider
discrete actions. To address the limitations, we propose a framework for the
full automation of the generation, execution, and evaluation of FCOG tasks. In
addition, we introduce an approach to solving the FCOG tasks by dividing them
into four distinct subtasks.
