  Synthesizing complex whole-body manipulation behaviors has fundamental
challenges due to the rapidly growing combinatorics inherent to contact
interaction planning. While model-based methods have shown promising results in
solving long-horizon manipulation tasks, they often work under strict
assumptions, such as known model parameters, oracular observation of the
environment state, and simplified dynamics, resulting in plans that cannot
easily transfer to hardware. Learning-based approaches, such as imitation
learning (IL) and reinforcement learning (RL), have been shown to be robust
when operating over in-distribution states; however, they need heavy human
supervision. Specifically, model-free RL requires a tedious reward-shaping
process. IL methods, on the other hand, rely on human demonstrations that
involve advanced teleoperation methods. In this work, we propose a plan-guided
reinforcement learning (PGRL) framework to combine the advantages of
model-based planning and reinforcement learning. Our method requires minimal
human supervision because it relies on plans generated by model-based planners
to guide the exploration in RL. In exchange, RL derives a more robust policy
thanks to domain randomization. We test this approach on a whole-body
manipulation task on Punyo, an upper-body humanoid robot with compliant,
air-filled arm coverings, to pivot and lift a large box. Our preliminary
results indicate that the proposed methodology is promising to address
challenges that remain difficult for either model- or learning-based strategies
alone.
