  This paper investigates the problem of interactively learning behaviors
communicated by a human teacher using positive and negative feedback. Much
previous work on this problem has made the assumption that people provide
feedback for decisions that is dependent on the behavior they are teaching and
is independent from the learner's current policy. We present empirical results
that show this assumption to be false -- whether human trainers give a positive
or negative feedback for a decision is influenced by the learner's current
policy. Based on this insight, we introduce {\em Convergent Actor-Critic by
Humans} (COACH), an algorithm for learning from policy-dependent feedback that
converges to a local optimum. Finally, we demonstrate that COACH can
successfully learn multiple behaviors on a physical robot.
