  Generating varied scenarios through simulation is crucial for training and
evaluating safety-critical systems, such as autonomous vehicles. Yet, the task
of modeling the trajectories of other vehicles to simulate diverse and
meaningful close interactions remains prohibitively costly. Adopting language
descriptions to generate driving behaviors emerges as a promising strategy,
offering a scalable and intuitive method for human operators to simulate a wide
range of driving interactions. However, the scarcity of large-scale annotated
language-trajectory data makes this approach challenging.
  To address this gap, we propose Text-to-Drive (T2D) to synthesize diverse
driving behaviors via Large Language Models (LLMs). We introduce a
knowledge-driven approach that operates in two stages. In the first stage, we
employ the embedded knowledge of LLMs to generate diverse language descriptions
of driving behaviors for a scene. Then, we leverage LLM's reasoning
capabilities to synthesize these behaviors in simulation. At its core, T2D
employs an LLM to construct a state chart that maps low-level states to
high-level abstractions. This strategy aids in downstream tasks such as
summarizing low-level observations, assessing policy alignment with behavior
description, and shaping the auxiliary reward, all without needing human
supervision. With our knowledge-driven approach, we demonstrate that T2D
generates more diverse trajectories compared to other baselines and offers a
natural language interface that allows for interactive incorporation of human
preference. Please check our website for more examples:
https://text-to-drive.github.io/
