  Developing a suitable Deep Neural Network (DNN) often requires significant
iteration, where different model versions are evaluated and compared. While
metrics such as accuracy are a powerful means to succinctly describe a model's
performance across a dataset or to directly compare model versions,
practitioners often wish to gain a deeper understanding of the factors that
influence a model's predictions. Interpretability techniques such as
gradient-based methods and local approximations can be used to examine small
sets of inputs in fine detail, but it can be hard to determine if results from
small sets generalize across a dataset. We introduce IMACS, a method that
combines gradient-based model attributions with aggregation and visualization
techniques to summarize differences in attributions between two DNN image
models. More specifically, IMACS extracts salient input features from an
evaluation dataset, clusters them based on similarity, then visualizes
differences in model attributions for similar input features. In this work, we
introduce a framework for aggregating, summarizing, and comparing the
attribution information for two models across a dataset; present visualizations
that highlight differences between 2 image classification models; and show how
our technique can uncover behavioral differences caused by domain shift between
two models trained on satellite images.
