  Recent machine learning approaches have been effective in Artificial
Intelligence (AI) applications. They produce robust results with a high level
of accuracy. However, most of these techniques do not provide
human-understandable explanations for supporting their results and decisions.
They usually act as black boxes, and it is not easy to understand how decisions
have been made. Explainable Artificial Intelligence (XAI), which has received
much interest recently, tries to provide human-understandable explanations for
decision-making and trained AI models. For instance, in digital agriculture,
related domains often present peculiar or input features with no link to
background knowledge. The application of the data mining process on
agricultural data leads to results (knowledge), which are difficult to explain.
In this paper, we propose a knowledge map model and an ontology design as an
XAI framework (OAK4XAI) to deal with this issue. The framework does not only
consider the data analysis part of the process, but it takes into account the
semantics aspect of the domain knowledge via an ontology and a knowledge map
model, provided as modules of the framework. Many ongoing XAI studies aim to
provide accurate and verbalizable accounts for how given feature values
contribute to model decisions. The proposed approach, however, focuses on
providing consistent information and definitions of concepts, algorithms, and
values involved in the data mining models. We built an Agriculture Computing
Ontology (AgriComO) to explain the knowledge mined in agriculture. AgriComO has
a well-designed structure and includes a wide range of concepts and
transformations suitable for agriculture and computing domains.
