  While reinforcement learning (RL) has become a more popular approach for
robotics, designing sufficiently informative reward functions for complex tasks
has proven to be extremely difficult due their inability to capture human
intent and policy exploitation. Preference based RL algorithms seek to overcome
these challenges by directly learning reward functions from human feedback.
Unfortunately, prior work either requires an unreasonable number of queries
implausible for any human to answer or overly restricts the class of reward
functions to guarantee the elicitation of the most informative queries,
resulting in models that are insufficiently expressive for realistic robotics
tasks. Contrary to most works that focus on query selection to \emph{minimize}
the amount of data required for learning reward functions, we take an opposite
approach: \emph{expanding} the pool of available data by viewing
human-in-the-loop RL through the more flexible lens of multi-task learning.
Motivated by the success of meta-learning, we pre-train preference models on
prior task data and quickly adapt them for new tasks using only a handful of
queries. Empirically, we reduce the amount of online feedback needed to train
manipulation policies in Meta-World by 20$\times$, and demonstrate the
effectiveness of our method on a real Franka Panda Robot. Moreover, this
reduction in query-complexity allows us to train robot policies from actual
human users. Videos of our results and code can be found at
https://sites.google.com/view/few-shot-preference-rl/home.
