  Complex behaviors are often driven by an internal model, which integrates
sensory information over time and facilitates long-term planning. Inferring an
agent's internal model is a crucial ingredient in social interactions (theory
of mind), for imitation learning, and for interpreting neural activities of
behaving agents. Here we describe a generic method to model an agent's behavior
under an environment with uncertainty, and infer the agent's internal model,
reward function, and dynamic beliefs. We apply our method to a simulated agent
performing a naturalistic foraging task. We assume the agent behaves rationally
--- that is, they take actions that optimize their subjective utility according
to their understanding of the task and its relevant causal variables. We model
this rational solution as a Partially Observable Markov Decision Process
(POMDP) where the agent may make wrong assumptions about the task parameters.
Given the agent's sensory observations and actions, we learn its internal model
and reward function by maximum likelihood estimation over a set of
task-relevant parameters. The Markov property of the POMDP enables us to
characterize the transition probabilities between internal belief states and
iteratively estimate the agent's policy using a constrained
Expectation-Maximization (EM) algorithm. We validate our method on simulated
agents performing suboptimally on a foraging task currently used in many
neuroscience experiments, and successfully recover their internal model and
reward function. Our work lays a critical foundation to discover how the brain
represents and computes with dynamic beliefs.
