  Understanding and following natural language instructions while navigating
through complex, real-world environments poses a significant challenge for
general-purpose robots. These environments often include obstacles and
pedestrians, making it essential for autonomous agents to possess the
capability of self-corrected planning to adjust their actions based on feedback
from the surroundings. However, the majority of existing vision-and-language
navigation (VLN) methods primarily operate in less realistic simulator settings
and do not incorporate environmental feedback into their decision-making
processes. To address this gap, we introduce a novel zero-shot framework called
CorNav, utilizing a large language model for decision-making and comprising two
key components: 1) incorporating environmental feedback for refining future
plans and adjusting its actions, and 2) multiple domain experts for parsing
instructions, scene understanding, and refining predicted actions. In addition
to the framework, we develop a 3D simulator that renders realistic scenarios
using Unreal Engine 5. To evaluate the effectiveness and generalization of
navigation agents in a zero-shot multi-task setting, we create a benchmark
called NavBench. Extensive experiments demonstrate that CorNav consistently
outperforms all baselines by a significant margin across all tasks. On average,
CorNav achieves a success rate of 28.1\%, surpassing the best baseline's
performance of 20.5\%.
