  Explaining the predictions of opaque machine learning algorithms is an
important and challenging task, especially as complex models are increasingly
used to assist in high-stakes decisions such as those arising in healthcare and
finance. Most popular tools for post-hoc explainable artificial intelligence
(XAI) are either insensitive to context (e.g., feature attributions) or
difficult to summarize (e.g., counterfactuals). In this paper, I introduce
$\textit{rational Shapley values}$, a novel XAI method that synthesizes and
extends these seemingly incompatible approaches in a rigorous, flexible manner.
I leverage tools from decision theory and causal modeling to formalize and
implement a pragmatic approach that resolves a number of known challenges in
XAI. By pairing the distribution of random variables with the appropriate
reference class for a given explanation task, I illustrate through theory and
experiments how user goals and knowledge can inform and constrain the solution
set in an iterative fashion. The method compares favorably to state of the art
XAI tools in a range of quantitative and qualitative comparisons.
