  State-of-the-art reinforcement learning (RL) algorithms suffer from high
sample complexity, particularly in the sparse reward case. A popular strategy
for mitigating this problem is to learn control policies by imitating a set of
expert demonstrations. The drawback of such approaches is that an expert needs
to produce demonstrations, which may be costly in practice. To address this
shortcoming, we propose Probabilistic Planning for Demonstration Discovery
(P2D2), a technique for automatically discovering demonstrations without access
to an expert. We formulate discovering demonstrations as a search problem and
leverage widely-used planning algorithms such as Rapidly-exploring Random Tree
to find demonstration trajectories. These demonstrations are used to initialize
a policy, then refined by a generic RL algorithm. We provide theoretical
guarantees of P2D2 finding successful trajectories, as well as bounds for its
sampling complexity. We experimentally demonstrate the method outperforms
classic and intrinsic exploration RL techniques in a range of classic control
and robotics tasks, requiring only a fraction of exploration samples and
achieving better asymptotic performance.
