  Robots can learn to imitate humans by inferring what the human is optimizing
for. One common framework for this is Bayesian reward learning, where the robot
treats the human's demonstrations and corrections as observations of their
underlying reward function. Unfortunately, this inference is
doubly-intractable: the robot must reason over all the trajectories the person
could have provided and all the rewards the person could have in mind. Prior
work uses existing robotic tools to approximate this normalizer. In this paper,
we group previous approaches into three fundamental classes and analyze the
theoretical pros and cons of their approach. We then leverage recent research
from the statistics community to introduce Double MH reward learning, a Monte
Carlo method for asymptotically learning the human's reward in continuous
spaces. We extend Double MH to conditionally independent settings (where each
human correction is viewed as completely separate) and conditionally dependent
environments (where the human's current correction may build on previous
inputs). Across simulations and user studies, our proposed approach infers the
human's reward parameters more accurately than the alternate approximations
when learning from either demonstrations or corrections. See videos here:
https://youtu.be/EkmT3o5K5ko
