  Recent years have witnessed an emerging paradigm shift toward embodied
artificial intelligence, in which an agent must learn to solve challenging
tasks by interacting with its environment. There are several challenges in
solving embodied multimodal tasks, including long-horizon planning,
vision-and-language grounding, and efficient exploration. We focus on a
critical bottleneck, namely the performance of planning and navigation. To
tackle this challenge, we propose a Neural SLAM approach that, for the first
time, utilizes several modalities for exploration, predicts an affordance-aware
semantic map, and plans over it at the same time. This significantly improves
exploration efficiency, leads to robust long-horizon planning, and enables
effective vision-and-language grounding. With the proposed Affordance-aware
Multimodal Neural SLAM (AMSLAM) approach, we obtain more than 40% improvement
over prior published work on the ALFRED benchmark and set a new
state-of-the-art generalization performance at a success rate of 23.48% on the
test unseen scenes.
