  Reinforcement learning and symbolic planning have both been used to build
intelligent autonomous agents. Reinforcement learning relies on learning from
interactions with real world, which often requires an unfeasibly large amount
of experience. Symbolic planning relies on manually crafted symbolic knowledge,
which may not be robust to domain uncertainties and changes. In this paper we
present a unified framework {\em PEORL} that integrates symbolic planning with
hierarchical reinforcement learning (HRL) to cope with decision-making in a
dynamic environment with uncertainties.
  Symbolic plans are used to guide the agent's task execution and learning, and
the learned experience is fed back to symbolic knowledge to improve planning.
This method leads to rapid policy search and robust symbolic plans in complex
domains. The framework is tested on benchmark domains of HRL.
