  Learning control policies to perform complex robotics tasks from human
preference data presents significant challenges. On the one hand, the
complexity of such tasks typically requires learning policies to perform a
variety of subtasks, then combining them to achieve the overall goal. At the
same time, comprehensive, well-engineered reward functions are typically
unavailable in such problems, while limited human preference data often is;
making efficient use of such data to guide learning is therefore essential.
Methods for learning to perform complex robotics tasks from human preference
data must overcome both these challenges simultaneously. In this work, we
introduce DIPPER: Direct Preference Optimization to Accelerate
Primitive-Enabled Hierarchical Reinforcement Learning, an efficient
hierarchical approach that leverages direct preference optimization to learn a
higher-level policy and reinforcement learning to learn a lower-level policy.
DIPPER enjoys improved computational efficiency due to its use of direct
preference optimization instead of standard preference-based approaches such as
reinforcement learning from human feedback, while it also mitigates the
well-known hierarchical reinforcement learning issues of non-stationarity and
infeasible subgoal generation due to our use of primitive-informed
regularization inspired by a novel bi-level optimization formulation of the
hierarchical reinforcement learning problem. To validate our approach, we
perform extensive experimental analysis on a variety of challenging robotics
tasks, demonstrating that DIPPER outperforms hierarchical and non-hierarchical
baselines, while ameliorating the non-stationarity and infeasible subgoal
generation issues of hierarchical reinforcement learning.
