  We consider the problem of explaining a class of tractable deep probabilistic
models, the Sum-Product Networks (SPNs) and present an algorithm ExSPN to
generate explanations. To this effect, we define the notion of a
context-specific independence tree(CSI-tree) and present an iterative algorithm
that converts an SPN to a CSI-tree. The resulting CSI-tree is both
interpretable and explainable to the domain expert. We achieve this by
extracting the conditional independencies encoded by the SPN and approximating
the local context specified by the structure of the SPN. Our extensive
empirical evaluations on synthetic, standard, and real-world clinical data sets
demonstrate that the CSI-tree exhibits superior explainability.
