  Humans are able to understand and perform complex tasks by strategically
structuring the tasks into incremental steps or subgoals. For a robot
attempting to learn to perform a sequential task with critical subgoal states,
such states can provide a natural opportunity for interaction with a human
expert. This paper analyzes the benefit of incorporating a notion of subgoals
into Inverse Reinforcement Learning (IRL) with a Human-In-The-Loop (HITL)
framework. The learning process is interactive, with a human expert first
providing input in the form of full demonstrations along with some subgoal
states. These subgoal states define a set of subtasks for the learning agent to
complete in order to achieve the final goal. The learning agent queries for
partial demonstrations corresponding to each subtask as needed when the agent
struggles with the subtask. The proposed Human Interactive IRL (HI-IRL)
framework is evaluated on several discrete path-planning tasks. We demonstrate
that subgoal-based interactive structuring of the learning task results in
significantly more efficient learning, requiring only a fraction of the
demonstration data needed for learning the underlying reward function with the
baseline IRL model.
