  We investigate the effect of including domain knowledge about a robotic
system's causal relations when generating explanations. To this end, we compare
two methods from explainable artificial intelligence, the popular KernelSHAP
and the recent causal SHAP, on a deep neural network trained using deep
reinforcement learning on the task of controlling a lever using a robotic
manipulator. A primary disadvantage of KernelSHAP is that its explanations
represent only the features' direct effects on a model's output, not
considering the indirect effects a feature can have on the output by affecting
other features. Causal SHAP uses a partial causal ordering to alter
KernelSHAP's sampling procedure to incorporate these indirect effects. This
partial causal ordering defines the causal relations between the features, and
we specify this using domain knowledge about the lever control task. We show
that enabling an explanation method to account for indirect effects and
incorporating some domain knowledge can lead to explanations that better agree
with human intuition. This is especially favorable for a real-world robotics
task, where there is considerable causality at play, and in addition, the
required domain knowledge is often handily available.
