  Reinforcement Learning (RL) plays a crucial role in advancing autonomous
driving technologies by maximizing reward functions to achieve the optimal
policy. However, crafting these reward functions has been a complex, manual
process in many practices. To reduce this complexity, we introduce a novel
framework that integrates Large Language Models (LLMs) with RL to improve
reward function design in autonomous driving. This framework utilizes the
coding capabilities of LLMs, proven in other areas, to generate and evolve
reward functions for highway scenarios. The framework starts with instructing
LLMs to create an initial reward function code based on the driving environment
and task descriptions. This code is then refined through iterative cycles
involving RL training and LLMs' reflection, which benefits from their ability
to review and improve the output. We have also developed a specific prompt
template to improve LLMs' understanding of complex driving simulations,
ensuring the generation of effective and error-free code. Our experiments in a
highway driving simulator across three traffic configurations show that our
method surpasses expert handcrafted reward functions, achieving a 22% higher
average success rate. This not only indicates safer driving but also suggests
significant gains in development productivity.
