  eXplainable AI focuses on generating explanations for the output of an AI
algorithm to a user, usually a decision-maker. Such user needs to interpret the
AI system in order to decide whether to trust the machine outcome. When
addressing this challenge, therefore, proper attention should be given to
produce explanations that are interpretable by the target community of users.
In this chapter, we claim for the need to better investigate what constitutes a
human explanation, i.e. a justification of the machine behaviour that is
interpretable and actionable by the human decision makers. In particular, we
focus on the contributions that Human Intelligence can bring to eXplainable AI,
especially in conjunction with the exploitation of Knowledge Graphs. Indeed, we
call for a better interplay between Knowledge Representation and Reasoning,
Social Sciences, Human Computation and Human-Machine Cooperation research -- as
already explored in other AI branches -- in order to support the goal of
eXplainable AI with the adoption of a Human-in-the-Loop approach.
