  Generating human-like behavior on robots is a great challenge especially in
dexterous manipulation tasks with robotic hands. Scripting policies from
scratch is intractable due to the high-dimensional control space, and training
policies with reinforcement learning (RL) and manual reward engineering can
also be hard and lead to unnatural motions. Leveraging the recent progress on
RL from Human Feedback, we propose a framework that learns a universal human
prior using direct human preference feedback over videos, for efficiently
tuning the RL policies on 20 dual-hand robot manipulation tasks in simulation,
without a single human demonstration. A task-agnostic reward model is trained
through iteratively generating diverse polices and collecting human preference
over the trajectories; it is then applied for regularizing the behavior of
polices in the fine-tuning stage. Our method empirically demonstrates more
human-like behaviors on robot hands in diverse tasks including even unseen
tasks, indicating its generalization capability.
