  Designing effective reward functions is crucial to training reinforcement
learning (RL) algorithms. However, this design is non-trivial, even for domain
experts, due to the subjective nature of certain tasks that are hard to
quantify explicitly. In recent works, large language models (LLMs) have been
used for reward generation from natural language task descriptions, leveraging
their extensive instruction tuning and commonsense understanding of human
behavior. In this work, we hypothesize that LLMs, guided by human feedback, can
be used to formulate human-aligned reward functions. Specifically, we study
this in the challenging setting of autonomous driving (AD), wherein notions of
"good" driving are tacit and hard to quantify. To this end, we introduce
REvolve, an evolutionary framework that uses LLMs for reward design in AD.
REvolve creates and refines reward functions by utilizing human feedback to
guide the evolution process, effectively translating implicit human knowledge
into explicit reward functions for training (deep) RL agents. We demonstrate
that agents trained on REvolve-designed rewards align closely with human
driving standards, thereby outperforming other state-of-the-art baselines.
