  Large language models (LLMs) have recently received considerable attention as
alternative solutions for task planning. However, comparing the performance of
language-oriented task planners becomes difficult, and there exists a dearth of
detailed exploration regarding the effects of various factors such as
pre-trained model selection and prompt construction. To address this, we
propose a benchmark system for automatically quantifying performance of task
planning for home-service embodied agents. Task planners are tested on two
pairs of datasets and simulators: 1) ALFRED and AI2-THOR, 2) an extension of
Watch-And-Help and VirtualHome. Using the proposed benchmark system, we perform
extensive experiments with LLMs and prompts, and explore several enhancements
of the baseline planner. We expect that the proposed benchmark tool would
accelerate the development of language-oriented task planners.
