  Reinforcement learning (RL) commonly assumes access to well-specified reward
functions, which many practical applications do not provide. Instead, recently,
more work has explored learning what to do from interacting with humans. So
far, most of these approaches model humans as being (nosily) rational and, in
particular, giving unbiased feedback. We argue that these models are too
simplistic and that RL researchers need to develop more realistic human models
to design and evaluate their algorithms. In particular, we argue that human
models have to be personal, contextual, and dynamic. This paper calls for
research from different disciplines to address key questions about how humans
provide feedback to AIs and how we can build more robust human-in-the-loop RL
systems.
