  Model-free Reinforcement Learning (RL) offers an attractive approach to learn
control policies for high-dimensional systems, but its relatively poor sample
complexity often forces training in simulated environments. Even in simulation,
goal-directed tasks whose natural reward function is sparse remain intractable
for state-of-the-art model-free algorithms for continuous control. The
bottleneck in these tasks is the prohibitive amount of exploration required to
obtain a learning signal from the initial state of the system. In this work, we
leverage physical priors in the form of an approximate system dynamics model to
design a curriculum scheme for a model-free policy optimization algorithm. Our
Backward Reachability Curriculum (BaRC) begins policy training from states that
require a small number of actions to accomplish the task, and expands the
initial state distribution backwards in a dynamically-consistent manner once
the policy optimization algorithm demonstrates sufficient performance. BaRC is
general, in that it can accelerate training of any model-free RL algorithm on a
broad class of goal-directed continuous control MDPs. Its curriculum strategy
is physically intuitive, easy-to-tune, and allows incorporating physical priors
to accelerate training without hindering the performance, flexibility, and
applicability of the model-free RL algorithm. We evaluate our approach on two
representative dynamic robotic learning problems and find substantial
performance improvement relative to previous curriculum generation techniques
and naive exploration strategies.
