  The exploration-exploitation trade-off is at the heart of reinforcement
learning (RL). However, most continuous control benchmarks used in recent RL
research only require local exploration. This led to the development of
algorithms that have basic exploration capabilities, and behave poorly in
benchmarks that require more versatile exploration. For instance, as
demonstrated in our empirical study, state-of-the-art RL algorithms such as
DDPG and TD3 are unable to steer a point mass in even small 2D mazes. In this
paper, we propose a new algorithm called "Plan, Backplay, Chain Skills" (PBCS)
that combines motion planning and reinforcement learning to solve hard
exploration environments. In a first phase, a motion planning algorithm is used
to find a single good trajectory, then an RL algorithm is trained using a
curriculum derived from the trajectory, by combining a variant of the Backplay
algorithm and skill chaining. We show that this method outperforms
state-of-the-art RL algorithms in 2D maze environments of various sizes, and is
able to improve on the trajectory obtained by the motion planning phase.
