  Preference-based reinforcement learning (RL) algorithms help avoid the
pitfalls of hand-crafted reward functions by distilling them from human
preference feedback, but they remain impractical due to the burdensome number
of labels required from the human, even for relatively simple tasks. In this
work, we demonstrate that encoding environment dynamics in the reward function
(REED) dramatically reduces the number of preference labels required in
state-of-the-art preference-based RL frameworks. We hypothesize that REED-based
methods better partition the state-action space and facilitate generalization
to state-action pairs not included in the preference dataset. REED iterates
between encoding environment dynamics in a state-action representation via a
self-supervised temporal consistency task, and bootstrapping the
preference-based reward function from the state-action representation. Whereas
prior approaches train only on the preference-labelled trajectory pairs, REED
exposes the state-action representation to all transitions experienced during
policy training. We explore the benefits of REED within the PrefPPO [1] and
PEBBLE [2] preference learning frameworks and demonstrate improvements across
experimental conditions to both the speed of policy learning and the final
policy performance. For example, on quadruped-walk and walker-walk with 50
preference labels, REED-based reward functions recover 83% and 66% of ground
truth reward policy performance and without REED only 38\% and 21\% are
recovered. For some domains, REED-based reward functions result in policies
that outperform policies trained on the ground truth reward.
