  Planning methods can solve temporally extended sequential decision making
problems by composing simple behaviors. However, planning requires suitable
abstractions for the states and transitions, which typically need to be
designed by hand. In contrast, model-free reinforcement learning (RL) can
acquire behaviors from low-level inputs directly, but often struggles with
temporally extended tasks. Can we utilize reinforcement learning to
automatically form the abstractions needed for planning, thus obtaining the
best of both approaches? We show that goal-conditioned policies learned with RL
can be incorporated into planning, so that a planner can focus on which states
to reach, rather than how those states are reached. However, with complex state
observations such as images, not all inputs represent valid states. We
therefore also propose using a latent variable model to compactly represent the
set of valid states for the planner, so that the policies provide an
abstraction of actions, and the latent variable model provides an abstraction
of states. We compare our method with planning-based and model-free methods and
find that our method significantly outperforms prior work when evaluated on
image-based robot navigation and manipulation tasks that require non-greedy,
multi-staged behavior.
