  Deep convolutional networks have been quite successful at various image
classification tasks. The current methods to explain the predictions of a
pre-trained model rely on gradient information, often resulting in saliency
maps that focus on the foreground object as a whole. However, humans typically
reason by dissecting an image and pointing out the presence of smaller
concepts. The final output is often an aggregation of the presence or absence
of these smaller concepts. In this work, we propose MACE: a Model Agnostic
Concept Extractor, which can explain the working of a convolutional network
through smaller concepts. The MACE framework dissects the feature maps
generated by a convolution network for an image to extract concept based
prototypical explanations. Further, it estimates the relevance of the extracted
concepts to the pre-trained model's predictions, a critical aspect required for
explaining the individual class predictions, missing in existing approaches. We
validate our framework using VGG16 and ResNet50 CNN architectures, and on
datasets like Animals With Attributes 2 (AWA2) and Places365. Our experiments
demonstrate that the concepts extracted by the MACE framework increase the
human interpretability of the explanations, and are faithful to the underlying
pre-trained black-box model.
