  Training automated agents to complete complex tasks in interactive
environments is challenging: reinforcement learning requires careful
hand-engineering of reward functions, imitation learning requires specialized
infrastructure and access to a human expert, and learning from intermediate
forms of supervision (like binary preferences) is time-consuming and extracts
little information from each human intervention. Can we overcome these
challenges by building agents that learn from rich, interactive feedback
instead? We propose a new supervision paradigm for interactive learning based
on "teachable" decision-making systems that learn from structured advice
provided by an external teacher. We begin by formalizing a class of
human-in-the-loop decision making problems in which multiple forms of
teacher-provided advice are available to a learner. We then describe a simple
learning algorithm for these problems that first learns to interpret advice,
then learns from advice to complete tasks even in the absence of human
supervision. In puzzle-solving, navigation, and locomotion domains, we show
that agents that learn from advice can acquire new skills with significantly
less human supervision than standard reinforcement learning algorithms and
often less than imitation learning.
