  Embodied Instruction Following (EIF) studies how autonomous mobile
manipulation robots should be controlled to accomplish long-horizon tasks
described by natural language instructions. While much research on EIF is
conducted in simulators, the ultimate goal of the field is to deploy the agents
in real life. This is one of the reasons why recent methods have moved away
from training models end-to-end and take modular approaches, which do not need
the costly expert operation data. However, as it is still in the early days of
importing modular ideas to EIF, a search for modules effective in the EIF task
is still far from a conclusion. In this paper, we propose to extend the modular
design using knowledge obtained from two external sources. First, we show that
embedding the physical constraints of the deployed robots into the module
design is highly effective. Our design also allows the same modular system to
work across robots of different configurations with minimal modifications.
Second, we show that the landmark-based object search, previously implemented
by a trained model requiring a dedicated set of data, can be replaced by an
implementation that prompts pretrained large language models for
landmark-object relationships, eliminating the need for collecting dedicated
training data. Our proposed Prompter achieves 41.53\% and 45.32\% on the ALFRED
benchmark with high-level instructions only and step-by-step instructions,
respectively, significantly outperforming the previous state of the art by
5.46\% and 9.91\%.
