  Offline reinforcement learning has become one of the most practical RL
settings. A recent success story has been RLHF, offline preference-based RL
(PBRL) with preference from humans. However, most existing works on offline RL
focus on the standard setting with scalar reward feedback. It remains unknown
how to universally transfer the existing rich understanding of offline RL from
the reward-based to the preference-based setting. In this work, we propose a
general framework to bridge this gap. Our key insight is transforming
preference feedback to scalar rewards via optimal reward labeling (ORL), and
then any reward-based offline RL algorithms can be applied to the dataset with
the reward labels. We theoretically show the connection between several recent
PBRL techniques and our framework combined with specific offline RL algorithms
in terms of how they utilize the preference signals. By combining reward
labeling with different algorithms, our framework can lead to new and
potentially more efficient offline PBRL algorithms. We empirically test our
framework on preference datasets based on the standard D4RL benchmark. When
combined with a variety of efficient reward-based offline RL algorithms, the
learning result achieved under our framework is comparable to training the same
algorithm on the dataset with actual rewards in many cases and better than the
recent PBRL baselines in most cases.
