  This work is inspired by recent advances in hierarchical reinforcement
learning (HRL) (Barto and Mahadevan 2003; Hengst 2010), and improvements in
learning efficiency from heuristic-based subgoal selection, experience replay
(Lin 1993; Andrychowicz et al. 2017), and task-based curriculum learning
(Bengio et al. 2009; Zaremba and Sutskever 2014). We propose a new method to
integrate HRL, experience replay and effective subgoal selection through an
implicit curriculum design based on human expertise to support sample-efficient
learning and enhance interpretability of the agent's behavior. Human expertise
remains indispensable in many areas such as medicine (Buch, Ahmed, and
Maruthappu 2018) and law (Cath 2018), where interpretability, explainability
and transparency are crucial in the decision making process, for ethical and
legal reasons. Our method simplifies the complex task sets for achieving the
overall objectives by decomposing them into subgoals at different levels of
abstraction. Incorporating relevant subjective knowledge also significantly
reduces the computational resources spent in exploration for RL, especially in
high speed, changing, and complex environments where the transition dynamics
cannot be effectively learned and modelled in a short time. Experimental
results in two StarCraft II (SC2) (Vinyals et al. 2017) minigames demonstrate
that our method can achieve better sample efficiency than flat and end-to-end
RL methods, and provides an effective method for explaining the agent's
performance.
