  In this paper, we argue for a paradigm shift from the current model of
explainable artificial intelligence (XAI), which may be counter-productive to
better human decision making. In early decision support systems, we assumed
that we could give people recommendations and that they would consider them,
and then follow them when required. However, research found that people often
ignore recommendations because they do not trust them; or perhaps even worse,
people follow them blindly, even when the recommendations are wrong.
Explainable artificial intelligence mitigates this by helping people to
understand how and why models give certain recommendations. However, recent
research shows that people do not always engage with explainability tools
enough to help improve decision making. The assumption that people will engage
with recommendations and explanations has proven to be unfounded. We argue this
is because we have failed to account for two things. First, recommendations
(and their explanations) take control from human decision makers, limiting
their agency. Second, giving recommendations and explanations does not align
with the cognitive processes employed by people making decisions. This position
paper proposes a new conceptual framework called Evaluative AI for explainable
decision support. This is a machine-in-the-loop paradigm in which decision
support tools provide evidence for and against decisions made by people, rather
than provide recommendations to accept or reject. We argue that this mitigates
issues of over- and under-reliance on decision support tools, and better
leverages human expertise in decision making.
