  Building a generalist agent that can interact with the world is the
intriguing target of AI systems, thus spurring the research for embodied
navigation, where an agent is required to navigate according to instructions or
respond to queries. Despite the major progress attained, previous works
primarily focus on task-specific agents and lack generalizability to unseen
scenarios. Recently, LLMs have presented remarkable capabilities across various
fields, and provided a promising opportunity for embodied navigation. Drawing
on this, we propose the first generalist model for embodied navigation,
NaviLLM. It adapts LLMs to embodied navigation by introducing schema-based
instruction. The schema-based instruction flexibly casts various tasks into
generation problems, thereby unifying a wide range of tasks. This approach
allows us to integrate diverse data sources from various datasets into the
training, equipping NaviLLM with a wide range of capabilities required by
embodied navigation. We conduct extensive experiments to evaluate the
performance and generalizability of our model. The experimental results
demonstrate that our unified model achieves state-of-the-art performance on
CVDN, SOON, and ScanQA. Specifically, it surpasses the previous
stats-of-the-art method by a significant margin of 29% in goal progress on
CVDN. Moreover, our model also demonstrates strong generalizability and
presents impressive results on unseen tasks, e.g., embodied question answering
and 3D captioning.
