  Building interpretable parameterizations of real-world decision-making on the
basis of demonstrated behavior -- i.e. trajectories of observations and actions
made by an expert maximizing some unknown reward function -- is essential for
introspecting and auditing policies in different institutions. In this paper,
we propose learning explanations of expert decisions by modeling their reward
function in terms of preferences with respect to "what if" outcomes: Given the
current history of observations, what would happen if we took a particular
action? To learn these cost-benefit tradeoffs associated with the expert's
actions, we integrate counterfactual reasoning into batch inverse reinforcement
learning. This offers a principled way of defining reward functions and
explaining expert behavior, and also satisfies the constraints of real-world
decision-making -- where active experimentation is often impossible (e.g. in
healthcare). Additionally, by estimating the effects of different actions,
counterfactuals readily tackle the off-policy nature of policy evaluation in
the batch setting, and can naturally accommodate settings where the expert
policies depend on histories of observations rather than just current states.
Through illustrative experiments in both real and simulated medical
environments, we highlight the effectiveness of our batch, counterfactual
inverse reinforcement learning approach in recovering accurate and
interpretable descriptions of behavior.
