  We present a method for enabling Reinforcement Learning of motor control
policies for complex skills such as dexterous manipulation. We posit that a key
difficulty for training such policies is the difficulty of exploring the
problem state space, as the accessible and useful regions of this space form a
complex structure along manifolds of the original high-dimensional state space.
This work presents a method to enable and support exploration with
Sampling-based Planning. We use a generally applicable non-holonomic
Rapidly-exploring Random Trees algorithm and present multiple methods to use
the resulting structure to bootstrap model-free Reinforcement Learning. Our
method is effective at learning various challenging dexterous motor control
skills of higher difficulty than previously shown. In particular, we achieve
dexterous in-hand manipulation of complex objects while simultaneously securing
the object without the use of passive support surfaces. These policies also
transfer effectively to real robots. A number of example videos can also be
found on the project website: https://sbrl.cs.columbia.edu
