  Discovering useful temporal abstractions, in the form of options, is widely
thought to be key to applying reinforcement learning and planning to
increasingly complex domains. Building on the empirical success of the Expert
Iteration approach to policy learning used in AlphaZero, we propose Option
Iteration, an analogous approach to option discovery. Rather than learning a
single strong policy that is trained to match the search results everywhere,
Option Iteration learns a set of option policies trained such that for each
state encountered, at least one policy in the set matches the search results
for some horizon into the future. Intuitively, this may be significantly easier
as it allows the algorithm to hedge its bets compared to learning a single
globally strong policy, which may have complex dependencies on the details of
the current state. Having learned such a set of locally strong policies, we can
use them to guide the search algorithm resulting in a virtuous cycle where
better options lead to better search results which allows for training of
better options. We demonstrate experimentally that planning using options
learned with Option Iteration leads to a significant benefit in challenging
planning environments compared to an analogous planning algorithm operating in
the space of primitive actions and learning a single rollout policy with Expert
Iteration.
