  The goal of explainable Artificial Intelligence (XAI) is to generate
human-interpretable explanations, but there are no computationally precise
theories of how humans interpret AI generated explanations. The lack of theory
means that validation of XAI must be done empirically, on a case-by-case basis,
which prevents systematic theory-building in XAI. We propose a psychological
theory of how humans draw conclusions from saliency maps, the most common form
of XAI explanation, which for the first time allows for precise prediction of
explainee inference conditioned on explanation. Our theory posits that absent
explanation humans expect the AI to make similar decisions to themselves, and
that they interpret an explanation by comparison to the explanations they
themselves would give. Comparison is formalized via Shepard's universal law of
generalization in a similarity space, a classic theory from cognitive science.
A pre-registered user study on AI image classifications with saliency map
explanations demonstrate that our theory quantitatively matches participants'
predictions of the AI.
