  The complex nature of artificial neural networks raises concerns on their
reliability, trustworthiness, and fairness in real-world scenarios. The Shapley
value -- a solution concept from game theory -- is one of the most popular
explanation methods for machine learning models. More traditionally, from a
statistical perspective, feature importance is defined in terms of conditional
independence. So far, these two approaches to interpretability and feature
importance have been considered separate and distinct. In this work, we show
that Shapley-based explanation methods and conditional independence testing are
closely related. We introduce the SHAPley EXplanation Randomization Test
(SHAP-XRT), a testing procedure inspired by the Conditional Randomization Test
(CRT) for a specific notion of local (i.e., on a sample) conditional
independence. With it, we prove that for binary classification problems, the
marginal contributions in the Shapley value provide lower and upper bounds to
the expected $p$-values of their respective tests. Furthermore, we show that
the Shapley value itself provides an upper bound to the expected $p$-value of a
global (i.e., overall) null hypothesis. As a result, we further our
understanding of Shapley-based explanation methods from a novel perspective and
characterize the conditions under which one can make statistically valid claims
about feature importance via the Shapley value.
