  Identifying algorithms that flexibly and efficiently discover
temporally-extended multi-phase plans is an essential step for the advancement
of robotics and model-based reinforcement learning. The core problem of
long-range planning is finding an efficient way to search through the tree of
possible action sequences. Existing non-learned planning solutions from the
Task and Motion Planning (TAMP) literature rely on the existence of logical
descriptions for the effects and preconditions for actions. This constraint
allows TAMP methods to efficiently reduce the tree search problem but limits
their ability to generalize to unseen and complex physical environments. In
contrast, deep reinforcement learning (DRL) methods use flexible
neural-network-based function approximators to discover policies that
generalize naturally to unseen circumstances. However, DRL methods struggle to
handle the very sparse reward landscapes inherent to long-range multi-step
planning situations. Here, we propose the Curious Sample Planner (CSP), which
fuses elements of TAMP and DRL by combining a curiosity-guided sampling
strategy with imitation learning to accelerate planning. We show that CSP can
efficiently discover interesting and complex temporally-extended plans for
solving a wide range of physically realistic 3D tasks. In contrast, standard
planning and learning methods often fail to solve these tasks at all or do so
only with a huge and highly variable number of training samples. We explore the
use of a variety of curiosity metrics with CSP and analyze the types of
solutions that CSP discovers. Finally, we show that CSP supports task transfer
so that the exploration policies learned during experience with one task can
help improve efficiency on related tasks.
