  In explanatory interactive learning (XIL) the user queries the learner, then
the learner explains its answer to the user and finally the loop repeats. XIL
is attractive for two reasons, (1) the learner becomes better and (2) the
user's trust increases. For both reasons to hold, the learner's explanations
must be useful to the user and the user must be allowed to ask useful
questions. Ideally, both questions and explanations should be grounded in a
causal model since they avoid spurious fallacies. Ultimately, we seem to seek a
causal variant of XIL. The question part on the user's end we believe to be
solved since the user's mental model can provide the causal model. But how
would the learner provide causal explanations? In this work we show that
existing explanation methods are not guaranteed to be causal even when provided
with a Structural Causal Model (SCM). Specifically, we use the popular,
proclaimed causal explanation method CXPlain to illustrate how the generated
explanations leave open the question of truly causal explanations. Thus as a
step towards causal XIL, we propose a solution to the lack of causal
explanations. We solve this problem by deriving from first principles an
explanation method that makes full use of a given SCM, which we refer to as
SC$\textbf{E}$ ($\textbf{E}$ standing for explanation). Since SCEs make use of
structural information, any causal graph learner can now provide human-readable
explanations. We conduct several experiments including a user study with 22
participants to investigate the virtue of SCE as causal explanations of SCMs.
