  This paper explores leveraging large language models for map-free off-road
navigation using generative AI, reducing the need for traditional data
collection and annotation. We propose a method where a robot receives verbal
instructions, converted to text through Whisper, and a large language model
(LLM) model extracts landmarks, preferred terrains, and crucial adverbs
translated into speed settings for constrained navigation. A language-driven
semantic segmentation model generates text-based masks for identifying
landmarks and terrain types in images. By translating 2D image points to the
vehicle's motion plane using camera parameters, an MPC controller can guides
the vehicle towards the desired terrain. This approach enhances adaptation to
diverse environments and facilitates the use of high-level instructions for
navigating complex and challenging terrains.
