  We present Hierarchical Deep Q-Network (HDQfD) that took first place in the
MineRL competition. HDQfD works on imperfect demonstrations and utilizes the
hierarchical structure of expert trajectories. We introduce the procedure of
extracting an effective sequence of meta-actions and subgoals from
demonstration data. We present a structured task-dependent replay buffer and
adaptive prioritizing technique that allow the HDQfD agent to gradually erase
poor-quality expert data from the buffer. In this paper, we present the details
of the HDQfD algorithm and give the experimental results in the Minecraft
domain.
