  Embodied Instruction Following (EIF) is a task of planning a long sequence of
sub-goals given high-level natural language instructions, such as "Rinse a
slice of lettuce and place on the white table next to the fork". To
successfully execute these long-term horizon tasks, we argue that an agent must
consider its past, i.e., historical data, when making decisions in each step.
Nevertheless, recent approaches in EIF often neglects the knowledge from
historical data and also do not effectively utilize information across the
modalities. To this end, we propose History-Aware Planning based on Fused
Information (HAPFI), effectively leveraging the historical data from diverse
modalities that agents collect while interacting with the environment.
Specifically, HAPFI integrates multiple modalities, including historical RGB
observations, bounding boxes, sub-goals, and high-level instructions, by
effectively fusing modalities via our Mutually Attentive Fusion method. Through
experiments with diverse comparisons, we show that an agent utilizing
historical multi-modal information surpasses all the compared methods that
neglect the historical data in terms of action planning capability, enabling
the generation of well-informed action plans for the next step. Moreover, we
provided qualitative evidence highlighting the significance of leveraging
historical multi-modal data, particularly in scenarios where the agent
encounters intermediate failures, showcasing its robust re-planning
capabilities.
