  Hierarchical planners that produce interpretable and appropriate plans are
desired, especially in its application to supporting human decision making. In
the typical development of the hierarchical planners, higher-level planners and
symbol grounding functions are manually created, and this manual creation
requires much human effort. In this paper, we propose a framework that can
automatically refine symbol grounding functions and a high-level planner to
reduce human effort for designing these modules. In our framework, symbol
grounding and high-level planning, which are based on manually-designed
knowledge bases, are modeled with semi-Markov decision processes. A policy
gradient method is then applied to refine the modules, in which two terms for
updating the modules are considered. The first term, called a reinforcement
term, contributes to updating the modules to improve the overall performance of
a hierarchical planner to produce appropriate plans. The second term, called a
penalty term, contributes to keeping refined modules consistent with the
manually-designed original modules. Namely, it keeps the planner, which uses
the refined modules, producing interpretable plans. We perform preliminary
experiments to solve the Mountain car problem, and its results show that a
manually-designed high-level planner and symbol grounding function were
successfully refined by our framework.
