  We investigate a formalism for the conditions of a successful explanation of
AI. We consider "success" to depend not only on what information the
explanation contains, but also on what information the human explainee
understands from it. Theory of mind literature discusses the folk concepts that
humans use to understand and generalize behavior. We posit that folk concepts
of behavior provide us with a "language" that humans understand behavior with.
We use these folk concepts as a framework of social attribution by the human
explainee - the information constructs that humans are likely to comprehend
from explanations - by introducing a blueprint for an explanatory narrative
(Figure 1) that explains AI behavior with these constructs. We then demonstrate
that many XAI methods today can be mapped to folk concepts of behavior in a
qualitative evaluation. This allows us to uncover their failure modes that
prevent current methods from explaining successfully - i.e., the information
constructs that are missing for any given XAI method, and whose inclusion can
decrease the likelihood of misunderstanding AI behavior.
