  Predictive black-box models can exhibit high accuracy but their opaque nature
hinders their uptake in safety-critical deployment environments. Explanation
methods (XAI) can provide confidence for decision-making through increased
transparency. However, existing XAI methods are not tailored towards models in
sensitive domains where one predictor is of special interest, such as a
treatment effect in a clinical model, or ethnicity in policy models. We
introduce Path-Wise Shapley effects (PWSHAP), a framework for assessing the
targeted effect of a binary (e.g.~treatment) variable from a complex outcome
model. Our approach augments the predictive model with a user-defined directed
acyclic graph (DAG). The method then uses the graph alongside on-manifold
Shapley values to identify effects along causal pathways whilst maintaining
robustness to adversarial attacks. We establish error bounds for the identified
path-wise Shapley effects and for Shapley values. We show PWSHAP can perform
local bias and mediation analyses with faithfulness to the model. Further, if
the targeted variable is randomised we can quantify local effect modification.
We demonstrate the resolution, interpretability, and true locality of our
approach on examples and a real-world experiment.
