  Despite substantial progress in applying neural networks (NN) to a wide
variety of areas, they still largely suffer from a lack of transparency and
interpretability. While recent developments in explainable artificial
intelligence attempt to bridge this gap (e.g., by visualizing the correlation
between input pixels and final outputs), these approaches are limited to
explaining low-level relationships, and crucially, do not provide insights on
error correction. In this work, we propose a framework (VRX) to interpret
classification NNs with intuitive structural visual concepts. Given a trained
classification model, the proposed VRX extracts relevant class-specific visual
concepts and organizes them using structural concept graphs (SCG) based on
pairwise concept relationships. By means of knowledge distillation, we show VRX
can take a step towards mimicking the reasoning process of NNs and provide
logical, concept-level explanations for final model decisions. With extensive
experiments, we empirically show VRX can meaningfully answer "why" and "why
not" questions about the prediction, providing easy-to-understand insights
about the reasoning process. We also show that these insights can potentially
provide guidance on improving NN's performance.
