  Enabling embodied agents to complete complex human instructions from natural
language is crucial to autonomous systems in household services. Conventional
methods can only accomplish human instructions in the known environment where
all interactive objects are provided to the embodied agent, and directly
deploying the existing approaches for the unknown environment usually generates
infeasible plans that manipulate non-existing objects. On the contrary, we
propose an embodied instruction following (EIF) method for complex tasks in the
unknown environment, where the agent efficiently explores the unknown
environment to generate feasible plans with existing objects to accomplish
abstract instructions. Specifically, we build a hierarchical embodied
instruction following framework including the high-level task planner and the
low-level exploration controller with multimodal large language models. We then
construct a semantic representation map of the scene with dynamic region
attention to demonstrate the known visual clues, where the goal of task
planning and scene exploration is aligned for human instruction. For the task
planner, we generate the feasible step-by-step plans for human goal
accomplishment according to the task completion process and the known visual
clues. For the exploration controller, the optimal navigation or object
interaction policy is predicted based on the generated step-wise plans and the
known visual clues. The experimental results demonstrate that our method can
achieve 45.09% success rate in 204 complex human instructions such as making
breakfast and tidying rooms in large house-level scenes.
