  Offline preference-based reinforcement learning (RL), which focuses on
optimizing policies using human preferences between pairs of trajectory
segments selected from an offline dataset, has emerged as a practical avenue
for RL applications. Existing works rely on extracting step-wise reward signals
from trajectory-wise preference annotations, assuming that preferences
correlate with the cumulative Markovian rewards. However, such methods fail to
capture the holistic perspective of data annotation: Humans often assess the
desirability of a sequence of actions by considering the overall outcome rather
than the immediate rewards. To address this challenge, we propose to model
human preferences using rewards conditioned on future outcomes of the
trajectory segments, i.e. the hindsight information. For downstream RL
optimization, the reward of each step is calculated by marginalizing over
possible future outcomes, the distribution of which is approximated by a
variational auto-encoder trained using the offline dataset. Our proposed
method, Hindsight Preference Learning (HPL), can facilitate credit assignment
by taking full advantage of vast trajectory data available in massive unlabeled
datasets. Comprehensive empirical studies demonstrate the benefits of HPL in
delivering robust and advantageous rewards across various domains. Our code is
publicly released at https://github.com/typoverflow/WiseRL.
