  In this critical survey, we analyze typical claims on the relationship
between explainable AI (XAI) and fairness to disentangle the multidimensional
relationship between these two concepts. Based on a systematic literature
review and a subsequent qualitative content analysis, we identify seven
archetypal claims from 175 scientific articles on the alleged fairness benefits
of XAI. We present crucial caveats with respect to these claims and provide an
entry point for future discussions around the potentials and limitations of XAI
for specific fairness desiderata. Importantly, we notice that claims are often
(i) vague and simplistic, (ii) lacking normative grounding, or (iii) poorly
aligned with the actual capabilities of XAI. We suggest to conceive XAI not as
an ethical panacea but as one of many tools to approach the multidimensional,
sociotechnical challenge of algorithmic fairness. Moreover, when making a claim
about XAI and fairness, we emphasize the need to be more specific about what
kind of XAI method is used, which fairness desideratum it refers to, how
exactly it enables fairness, and who is the stakeholder that benefits from XAI.
