  An important use of machine learning is to learn what people value. What
posts or photos should a user be shown? Which jobs or activities would a person
find rewarding? In each case, observations of people's past choices can inform
our inferences about their likes and preferences. If we assume that choices are
approximately optimal according to some utility function, we can treat
preference inference as Bayesian inverse planning. That is, given a prior on
utility functions and some observed choices, we invert an optimal
decision-making process to infer a posterior distribution on utility functions.
However, people often deviate from approximate optimality. They have false
beliefs, their planning is sub-optimal, and their choices may be temporally
inconsistent due to hyperbolic discounting and other biases. We demonstrate how
to incorporate these deviations into algorithms for preference inference by
constructing generative models of planning for agents who are subject to false
beliefs and time inconsistency. We explore the inferences these models make
about preferences, beliefs, and biases. We present a behavioral experiment in
which human subjects perform preference inference given the same observations
of choices as our model. Results show that human subjects (like our model)
explain choices in terms of systematic deviations from optimal behavior and
suggest that they take such deviations into account when inferring preferences.
