  According to cognitive psychology and related disciplines, the development of
complex problem-solving behaviour in biological agents depends on hierarchical
cognitive mechanisms. Hierarchical reinforcement learning is a promising
computational approach that may eventually yield comparable problem-solving
behaviour in artificial agents and robots. However, to date the problem-solving
abilities of many human and non-human animals are clearly superior to those of
artificial systems. Here, we propose steps to integrate biologically inspired
hierarchical mechanisms to enable advanced problem-solving skills in artificial
agents. Therefore, we first review the literature in cognitive psychology to
highlight the importance of compositional abstraction and predictive
processing. Then we relate the gained insights with contemporary hierarchical
reinforcement learning methods. Interestingly, our results suggest that all
identified cognitive mechanisms have been implemented individually in isolated
computational architectures, raising the question of why there exists no single
unifying architecture that integrates them. As our final contribution, we
address this question by providing an integrative perspective on the
computational challenges to develop such a unifying architecture. We expect our
results to guide the development of more sophisticated cognitively inspired
hierarchical machine learning architectures.
