  Explaining the decisions of black-box models is a central theme in the study
of trustworthy ML. Numerous measures have been proposed in the literature;
however, none of them take an axiomatic approach to causal explainability. In
this work, we propose three explanation measures which aggregate the set of all
but-for causes -- a necessary and sufficient explanation -- into feature
importance weights. Our first measure is a natural adaptation of Chockler and
Halpern's notion of causal responsibility, whereas the other two correspond to
existing game-theoretic influence measures. We present an axiomatic treatment
for our proposed indices, showing that they can be uniquely characterized by a
set of desirable properties. We also extend our approach to derive a new method
to compute the Shapley-Shubik and Banzhaf indices for black-box model
explanations. Finally, we analyze and compare the necessity and sufficiency of
all our proposed explanation measures in practice using the Adult-Income
dataset. Thus, our work is the first to formally bridge the gap between model
explanations, game-theoretic influence, and causal analysis.
