  Learning reward functions remains the bottleneck to equip a robot with a
broad repertoire of skills. Large Language Models (LLM) contain valuable
task-related knowledge that can potentially aid in the learning of reward
functions. However, the proposed reward function can be imprecise, thus
ineffective which requires to be further grounded with environment information.
We proposed a method to learn rewards more efficiently in the absence of
humans. Our approach consists of two components: We first use the LLM to
propose features and parameterization of the reward, then update the parameters
through an iterative self-alignment process. In particular, the process
minimizes the ranking inconsistency between the LLM and the learnt reward
functions based on the execution feedback. The method was validated on 9 tasks
across 2 simulation environments. It demonstrates a consistent improvement over
training efficacy and efficiency, meanwhile consuming significantly fewer GPT
tokens compared to the alternative mutation-based method.
