  This paper evaluates whether training a decision tree based on concepts
extracted from a concept-based explainer can increase interpretability for
Convolutional Neural Networks (CNNs) models and boost the fidelity and
performance of the used explainer. CNNs for computer vision have shown
exceptional performance in critical industries. However, it is a significant
barrier when deploying CNNs due to their complexity and lack of
interpretability. Recent studies to explain computer vision models have shifted
from extracting low-level features (pixel-based explanations) to mid-or
high-level features (concept-based explanations). The current research
direction tends to use extracted features in developing approximation
algorithms such as linear or decision tree models to interpret an original
model. In this work, we modify one of the state-of-the-art concept-based
explanations and propose an alternative framework named TreeICE. We design a
systematic evaluation based on the requirements of fidelity (approximate models
to original model's labels), performance (approximate models to ground-truth
labels), and interpretability (meaningful of approximate models to humans). We
conduct computational evaluation (for fidelity and performance) and human
subject experiments (for interpretability) We find that Tree-ICE outperforms
the baseline in interpretability and generates more human readable explanations
in the form of a semantic tree structure. This work features how important to
have more understandable explanations when interpretability is crucial.
