  Deep reinforcement learning (RL) agents are able to learn contact-rich
manipulation tasks by maximizing a reward signal, but require large amounts of
experience, especially in environments with many obstacles that complicate
exploration. In contrast, motion planners use explicit models of the agent and
environment to plan collision-free paths to faraway goals, but suffer from
inaccurate models in tasks that require contacts with the environment. To
combine the benefits of both approaches, we propose motion planner augmented RL
(MoPA-RL) which augments the action space of an RL agent with the long-horizon
planning capabilities of motion planners. Based on the magnitude of the action,
our approach smoothly transitions between directly executing the action and
invoking a motion planner. We evaluate our approach on various simulated
manipulation tasks and compare it to alternative action spaces in terms of
learning efficiency and safety. The experiments demonstrate that MoPA-RL
increases learning efficiency, leads to a faster exploration, and results in
safer policies that avoid collisions with the environment. Videos and code are
available at https://clvrai.com/mopa-rl .
