  In preference-based Reinforcement Learning (RL), obtaining a large number of
preference labels are both time-consuming and costly. Furthermore, the queried
human preferences cannot be utilized for the new tasks. In this paper, we
propose Zero-shot Cross-task Preference Alignment and Robust Reward Learning
(PEARL), which learns policies from cross-task preference transfer without any
human labels of the target task. Our contributions include two novel components
that facilitate the transfer and learning process. The first is Cross-task
Preference Alignment (CPA), which transfers the preferences between tasks via
optimal transport. The key idea of CPA is to use Gromov-Wasserstein distance to
align the trajectories between tasks, and the solved optimal transport matrix
serves as the correspondence between trajectories. The target task preferences
are computed as the weighted sum of source task preference labels with the
correspondence as weights. Moreover, to ensure robust learning from these
transferred labels, we introduce Robust Reward Learning (RRL), which considers
both reward mean and uncertainty by modeling rewards as Gaussian distributions.
Empirical results on robotic manipulation tasks from Meta-World and Robomimic
demonstrate that our method is capable of transferring preference labels across
tasks accurately and then learns well-behaved policies. Notably, our approach
significantly exceeds existing methods when there are few human preferences.
The code and videos of our method are available at:
https://sites.google.com/view/pearl-preference.
