  Balancing exploration and exploitation remains a key challenge in
reinforcement learning (RL). State-of-the-art RL algorithms suffer from high
sample complexity, particularly in the sparse reward case, where they can do no
better than to explore in all directions until the first positive rewards are
found. To mitigate this, we propose Rapidly Randomly-exploring Reinforcement
Learning (R3L). We formulate exploration as a search problem and leverage
widely-used planning algorithms such as Rapidly-exploring Random Tree (RRT) to
find initial solutions. These solutions are used as demonstrations to
initialize a policy, then refined by a generic RL algorithm, leading to faster
and more stable convergence. We provide theoretical guarantees of R3L
exploration finding successful solutions, as well as bounds for its sampling
complexity. We experimentally demonstrate the method outperforms classic and
intrinsic exploration techniques, requiring only a fraction of exploration
samples and achieving better asymptotic performance.
