  Recent works have extended notions of feature importance to \emph{semantic
concepts} that are inherently interpretable to the users interacting with a
black-box predictive model. Yet, precise statistical guarantees, such as false
positive rate control, are needed to communicate findings transparently and to
avoid unintended consequences in real-world scenarios. In this paper, we
formalize the global (i.e., over a population) and local (i.e., for a sample)
statistical importance of semantic concepts for the predictions of opaque
models, by means of conditional independence, which allows for rigorous
testing. We use recent ideas of sequential kernelized testing (SKIT) to induce
a rank of importance across concepts, and showcase the effectiveness and
flexibility of our framework on synthetic datasets as well as on image
classification tasks using vision-language models such as CLIP.
