  Realizing when a model is right for a wrong reason is not trivial and
requires a significant effort by model developers. In some cases an input
salience method, which highlights the most important parts of the input, may
reveal problematic reasoning. But scrutinizing highlights over many data
instances is tedious and often infeasible. Furthermore, analyzing examples in
isolation does not reveal general patterns in the data or in the model's
behavior. In this paper we aim to address these issues and go from
understanding single examples to understanding entire datasets and models. The
methodology we propose is based on aggregated salience maps, to which we apply
clustering, nearest neighbor search and visualizations. Using this methodology
we address multiple distinct but common model developer needs by showing how
problematic data and model behavior can be identified and explained -- a
necessary first step for improving the model.
