  Our goal is for AI systems to correctly identify and act according to their
human user's objectives. Cooperative Inverse Reinforcement Learning (CIRL)
formalizes this value alignment problem as a two-player game between a human
and robot, in which only the human knows the parameters of the reward function:
the robot needs to learn them as the interaction unfolds. Previous work showed
that CIRL can be solved as a POMDP, but with an action space size exponential
in the size of the reward parameter space. In this work, we exploit a specific
property of CIRL---the human is a full information agent---to derive an
optimality-preserving modification to the standard Bellman update; this reduces
the complexity of the problem by an exponential factor and allows us to relax
CIRL's assumption of human rationality. We apply this update to a variety of
POMDP solvers and find that it enables us to scale CIRL to non-trivial
problems, with larger reward parameter spaces, and larger action spaces for
both robot and human. In solutions to these larger problems, the human exhibits
pedagogic (teaching) behavior, while the robot interprets it as such and
attains higher value for the human.
