  eXplainable Artificial Intelligence (XAI) aims at providing understandable
explanations of black box models. In this paper, we evaluate current XAI
methods by scoring them based on ground truth simulations and sensitivity
analysis. To this end, we used an Electric Arc Furnace (EAF) model to better
understand the limits and robustness characteristics of XAI methods such as
SHapley Additive exPlanations (SHAP), Local Interpretable Model-agnostic
Explanations (LIME), as well as Averaged Local Effects (ALE) or Smooth
Gradients (SG) in a highly topical setting. These XAI methods were applied to
various types of black-box models and then scored based on their correctness
compared to the ground-truth sensitivity of the data-generating processes using
a novel scoring evaluation methodology over a range of simulated additive
noise. The resulting evaluation shows that the capability of the Machine
Learning (ML) models to capture the process accurately is, indeed, coupled with
the correctness of the explainability of the underlying data-generating
process. We furthermore show the differences between XAI methods in their
ability to correctly predict the true sensitivity of the modeled industrial
process.
