  Human explanations of high-level decisions are often expressed in terms of
key concepts the decisions are based on. In this paper, we study such
concept-based explainability for Deep Neural Networks (DNNs). First, we define
the notion of completeness, which quantifies how sufficient a particular set of
concepts is in explaining a model's prediction behavior based on the assumption
that complete concept scores are sufficient statistics of the model prediction.
Next, we propose a concept discovery method that aims to infer a complete set
of concepts that are additionally encouraged to be interpretable, which
addresses the limitations of existing methods on concept explanations. To
define an importance score for each discovered concept, we adapt game-theoretic
notions to aggregate over sets and propose ConceptSHAP. Via proposed metrics
and user studies, on a synthetic dataset with apriori-known concept
explanations, as well as on real-world image and language datasets, we validate
the effectiveness of our method in finding concepts that are both complete in
explaining the decisions and interpretable. (The code is released at
https://github.com/chihkuanyeh/concept_exp)
