  An increasing ubiquity of machine learning (ML) motivates research on
algorithms to explain ML models and their predictions -- so-called eXplainable
Artificial Intelligence (XAI). Despite many survey papers and discussions, the
goals and capabilities of XAI algorithms are far from being well understood. We
argue that this is because of a problematic reasoning scheme in XAI literature:
XAI algorithms are said to complement ML models with desired properties, such
as "interpretability", or "explainability". These properties are in turn
assumed to contribute to a goal, like "trust" in an ML system. But most
properties lack precise definitions and their relationship to such goals is far
from obvious. The result is a reasoning scheme that obfuscates research results
and leaves an important question unanswered: What can one expect from XAI
algorithms? In this article, we clarify the goals and capabilities of XAI
algorithms from a concrete perspective: that of their users. Explaining ML
models is only necessary if users have questions about them. We show that users
can ask diverse questions, but that only one of them can be answered by current
XAI algorithms. Answering this core question can be trivial, difficult or even
impossible, depending on the ML application. Based on these insights, we
outline which capabilities policymakers, researchers and society can reasonably
expect from XAI algorithms.
