  Explainable Artificial Intelligence is critical in unraveling decision-making
processes in complex machine learning models. LIME (Local Interpretable
Model-agnostic Explanations) is a well-known XAI framework for image analysis.
It utilizes image segmentation to create features to identify relevant areas
for classification. Consequently, poor segmentation can compromise the
consistency of the explanation and undermine the importance of the segments,
affecting the overall interpretability. Addressing these challenges, we
introduce DSEG-LIME (Data-Driven Segmentation LIME), featuring: i) a
data-driven segmentation for human-recognized feature generation, and ii) a
hierarchical segmentation procedure through composition. We benchmark DSEG-LIME
on pre-trained models with images from the ImageNet dataset - scenarios without
domain-specific knowledge. The analysis includes a quantitative evaluation
using established XAI metrics, complemented by a qualitative assessment through
a user study. Our findings demonstrate that DSEG outperforms in most of the XAI
metrics and enhances the alignment of explanations with human-recognized
concepts, significantly improving interpretability. The code is available
under: https://github. com/patrick-knab/DSEG-LIME
