  Two common approaches to sequential decision-making are AI planning (AIP) and
reinforcement learning (RL). Each has strengths and weaknesses. AIP is
interpretable, easy to integrate with symbolic knowledge, and often efficient,
but requires an up-front logical domain specification and is sensitive to
noise; RL only requires specification of rewards and is robust to noise but is
sample inefficient and not easily supplied with external knowledge. We propose
an integrative approach that combines high-level planning with RL, retaining
interpretability, transfer, and efficiency, while allowing for robust learning
of the lower-level plan actions. Our approach defines options in hierarchical
reinforcement learning (HRL) from AIP operators by establishing a
correspondence between the state transition model of AI planning problem and
the abstract state transition system of a Markov Decision Process (MDP).
Options are learned by adding intrinsic rewards to encourage consistency
between the MDP and AIP transition models. We demonstrate the benefit of our
integrated approach by comparing the performance of RL and HRL algorithms in
both MiniGrid and N-rooms environments, showing the advantage of our method
over the existing ones.
