  Recent efforts to enable visual navigation using large language models have
mainly focused on developing complex prompt systems. These systems incorporate
instructions, observations, and history into massive text prompts, which are
then combined with pre-trained large language models to facilitate visual
navigation. In contrast, our approach aims to fine-tune large language models
for visual navigation without extensive prompt engineering. Our design involves
a simple text prompt, current observations, and a history collector model that
gathers information from previous observations as input. For output, our design
provides a probability distribution of possible actions that the agent can take
during navigation. We train our model using human demonstrations and collision
signals from the Habitat-Matterport 3D Dataset (HM3D). Experimental results
demonstrate that our method outperforms state-of-the-art behavior cloning
methods and effectively reduces collision rates.
