  Supervised imitation learning, also known as behavioral cloning, suffers from
distribution drift leading to failures during policy execution. One approach to
mitigate this issue is to allow an expert to correct the agent's actions during
task execution, based on the expert's determination that the agent has reached
a `point of no return.' The agent's policy is then retrained using this new
corrective data. This approach alone can enable high-performance agents to be
learned, but at a substantial cost: the expert must vigilantly observe
execution until the policy reaches a specified level of success, and even at
that point, there is no guarantee that the policy will always succeed. To
address these limitations, we present FIRE (Failure Identification to Reduce
Expert Burden in intervention-based learning), a system that can predict when a
running policy will fail, halt its execution, and request a correction from the
expert. Unlike existing approaches that learn only from expert data, our
approach learns from both expert and non-expert data, akin to adversarial
learning. We demonstrate experimentally for a series of challenging
manipulation tasks that our method is able to recognize state-action pairs that
lead to failures. This permits seamless integration into an intervention-based
learning system, where we show an order-of-magnitude gain in sample efficiency
compared with a state-of-the-art inverse reinforcement learning method and
dramatically improved performance over an equivalent amount of data learned
with behavioral cloning.
