  Robots will eventually be part of every household. It is thus critical to
enable algorithms to learn from and be guided by non-expert users. In this
paper, we bring a human in the loop, and enable a human teacher to give
feedback to a learning agent in the form of natural language. We argue that a
descriptive sentence can provide a much stronger learning signal than a numeric
reward in that it can easily point to where the mistakes are and how to correct
them. We focus on the problem of image captioning in which the quality of the
output can easily be judged by non-experts. We propose a hierarchical
phrase-based captioning model trained with policy gradients, and design a
feedback network that provides reward to the learner by conditioning on the
human-provided feedback. We show that by exploiting descriptive feedback our
model learns to perform better than when given independently written human
captions.
