  Specifying rewards for reinforcement learned (RL) agents is challenging.
Preference-based RL (PbRL) mitigates these challenges by inferring a reward
from feedback over sets of trajectories. However, the effectiveness of PbRL is
limited by the amount of feedback needed to reliably recover the structure of
the target reward. We present the PRIor Over Rewards (PRIOR) framework, which
incorporates priors about the structure of the reward function and the
preference feedback into the reward learning process. Imposing these priors as
soft constraints on the reward learning objective reduces the amount of
feedback required by half and improves overall reward recovery. Additionally,
we demonstrate that using an abstract state space for the computation of the
priors further improves the reward learning and the agent's performance.
