  Reward design in reinforcement learning (RL) is challenging since specifying
human notions of desired behavior may be difficult via reward functions or
require many expert demonstrations. Can we instead cheaply design rewards using
a natural language interface? This paper explores how to simplify reward design
by prompting a large language model (LLM) such as GPT-3 as a proxy reward
function, where the user provides a textual prompt containing a few examples
(few-shot) or a description (zero-shot) of the desired behavior. Our approach
leverages this proxy reward function in an RL framework. Specifically, users
specify a prompt once at the beginning of training. During training, the LLM
evaluates an RL agent's behavior against the desired behavior described by the
prompt and outputs a corresponding reward signal. The RL agent then uses this
reward to update its behavior. We evaluate whether our approach can train
agents aligned with user objectives in the Ultimatum Game, matrix games, and
the DealOrNoDeal negotiation task. In all three tasks, we show that RL agents
trained with our framework are well-aligned with the user's objectives and
outperform RL agents trained with reward functions learned via supervised
learning
