  Explainable AI (XAI) aims to address the human need for safe and reliable AI
systems. However, numerous surveys emphasize the absence of a sound
mathematical formalization of key XAI notions -- remarkably including the term
"explanation" which still lacks a precise definition. To bridge this gap, this
paper presents the first mathematically rigorous definitions of key XAI notions
and processes, using the well-funded formalism of Category theory. We show that
our categorical framework allows to: (i) model existing learning schemes and
architectures, (ii) formally define the term "explanation", (iii) establish a
theoretical basis for XAI taxonomies, and (iv) analyze commonly overlooked
aspects of explaining methods. As a consequence, our categorical framework
promotes the ethical and secure deployment of AI technologies as it represents
a significant step towards a sound theoretical foundation of explainable AI.
