  Counterfactual explanations (CEs) are methods for generating an alternative
scenario that produces a different desirable outcome. For example, if a student
is predicted to fail a course, then counterfactual explanations can provide the
student with alternate ways so that they would be predicted to pass. The
applications are many. However, CEs are currently generated from machine
learning models that do not necessarily take into account the true causal
structure in the data. By doing this, bias can be introduced into the CE
quantities. I propose in this study to test the CEs using Judea Pearl's method
of computing counterfactuals which has thus far, surprisingly, not been seen in
the counterfactual explanation (CE) literature. I furthermore evaluate these
CEs on three different causal structures to show how the true underlying causal
structure affects the CEs that are generated. This study presented a method of
evaluating CEs using Pearl's method and it showed, (although using a limited
sample size), that thirty percent of the CEs conflicted with those computed by
Pearl's method. This shows that we cannot simply trust CEs and it is vital for
us to know the true causal structure before we blindly compute counterfactuals
using the original machine learning model.
