  Hierarchical reinforcement learning (HRL) has the potential to solve complex
long horizon tasks using temporal abstraction and increased exploration.
However, hierarchical agents are difficult to train due to inherent
non-stationarity. We present primitive enabled adaptive relabeling (PEAR), a
two-phase approach where we first perform adaptive relabeling on a few expert
demonstrations to generate efficient subgoal supervision, and then jointly
optimize HRL agents by employing reinforcement learning (RL) and imitation
learning (IL). We perform theoretical analysis to $(i)$ bound the
sub-optimality of our approach, and $(ii)$ derive a generalized plug-and-play
framework for joint optimization using RL and IL. Since PEAR utilizes only a
handful of expert demonstrations and considers minimal limiting assumptions on
the task structure, it can be easily integrated with typical off-policy RL
algorithms to produce a practical HRL approach. We perform extensive
experiments on challenging environments and show that PEAR is able to
outperform various hierarchical and non-hierarchical baselines on complex tasks
that require long term decision making. We also perform ablations to thoroughly
analyse the importance of our various design choices. Finally, we perform real
world robotic experiments on complex tasks and demonstrate that PEAR
consistently outperforms the baselines.
