  This study investigates the impact of machine learning models on the
generation of counterfactual explanations by conducting a benchmark evaluation
over three different types of models: a decision tree (fully transparent,
interpretable, white-box model), a random forest (semi-interpretable, grey-box
model), and a neural network (fully opaque, black-box model). We tested the
counterfactual generation process using four algorithms (DiCE, WatcherCF,
prototype, and GrowingSpheresCF) in the literature in 25 different datasets.
Our findings indicate that: (1) Different machine learning models have little
impact on the generation of counterfactual explanations; (2) Counterfactual
algorithms based uniquely on proximity loss functions are not actionable and
will not provide meaningful explanations; (3) One cannot have meaningful
evaluation results without guaranteeing plausibility in the counterfactual
generation. Algorithms that do not consider plausibility in their internal
mechanisms will lead to biased and unreliable conclusions if evaluated with the
current state-of-the-art metrics; (4) A counterfactual inspection analysis is
strongly recommended to ensure a robust examination of counterfactual
explanations and the potential identification of biases.
