  Inferring a person's goal from their behavior is an important problem in
applications of AI (e.g. automated assistants, recommender systems). The
workhorse model for this task is the rational actor model - this amounts to
assuming that people have stable reward functions, discount the future
exponentially, and construct optimal plans. Under the rational actor assumption
techniques such as inverse reinforcement learning (IRL) can be used to infer a
person's goals from their actions. A competing model is the dual-system model.
Here decisions are the result of an interplay between a fast, automatic,
heuristic-based system 1 and a slower, deliberate, calculating system 2. We
generalize the dual system framework to the case of Markov decision problems
and show how to compute optimal plans for dual-system agents. We show that
dual-system agents exhibit behaviors that are incompatible with rational actor
assumption. We show that naive applications of rational-actor IRL to the
behavior of dual-system agents can generate wrong inference about the agents'
goals and suggest interventions that actually reduce the agent's overall
utility. Finally, we adapt a simple IRL algorithm to correctly infer the goals
of dual system decision-makers. This allows us to make interventions that help,
rather than hinder, the dual-system agent's ability to reach their true goals.
