  In order to solve a task using reinforcement learning, it is necessary to
first formalise the goal of that task as a reward function. However, for many
real-world tasks, it is very difficult to manually specify a reward function
that never incentivises undesirable behaviour. As a result, it is increasingly
popular to use \emph{reward learning algorithms}, which attempt to \emph{learn}
a reward function from data. However, the theoretical foundations of reward
learning are not yet well-developed. In particular, it is typically not known
when a given reward learning algorithm with high probability will learn a
reward function that is safe to optimise. This means that reward learning
algorithms generally must be evaluated empirically, which is expensive, and
that their failure modes are difficult to anticipate in advance. One of the
roadblocks to deriving better theoretical guarantees is the lack of good
methods for quantifying the difference between reward functions. In this paper
we provide a solution to this problem, in the form of a class of pseudometrics
on the space of all reward functions that we call STARC (STAndardised Reward
Comparison) metrics. We show that STARC metrics induce both an upper and a
lower bound on worst-case regret, which implies that our metrics are tight, and
that any metric with the same properties must be bilipschitz equivalent to
ours. Moreover, we also identify a number of issues with reward metrics
proposed by earlier works. Finally, we evaluate our metrics empirically, to
demonstrate their practical efficacy. STARC metrics can be used to make both
theoretical and empirical analysis of reward learning algorithms both easier
and more principled.
