Searching for Plannable Domains can Speed up Reinforcement Learning
  Reinforcement learning (RL) involves sequential decision making in uncertain
environments. The aim of the decision-making agent is to maximize the benefit
of acting in its environment over an extended period of time. Finding an
optimal policy in RL may be very slow. To speed up learning, one often used
solution is the integration of planning, for example, Sutton's Dyna algorithm,
or various other methods using macro-actions.
  Here we suggest to separate plannable, i.e., close to deterministic parts of
the world, and focus planning efforts in this domain. A novel reinforcement
learning method called plannable RL (pRL) is proposed here. pRL builds a simple
model, which is used to search for macro actions. The simplicity of the model
makes planning computationally inexpensive. It is shown that pRL finds an
optimal policy, and that plannable macro actions found by pRL are near-optimal.
In turn, it is unnecessary to try large numbers of macro actions, which enables
fast learning. The utility of pRL is demonstrated by computer simulations.
