  We aim to explain a black-box classifier with the form: `data X is classified
as class Y because X \textit{has} A, B and \textit{does not have} C' in which
A, B, and C are high-level concepts. The challenge is that we have to discover
in an unsupervised manner a set of concepts, i.e., A, B and C, that is useful
for the explaining the classifier. We first introduce a structural generative
model that is suitable to express and discover such concepts. We then propose a
learning process that simultaneously learns the data distribution and
encourages certain concepts to have a large causal influence on the classifier
output. Our method also allows easy integration of user's prior knowledge to
induce high interpretability of concepts. Using multiple datasets, we
demonstrate that our method can discover useful binary concepts for
explanation.
