  We propose a novel method to explain trained deep neural networks (DNNs), by
distilling them into surrogate models using unsupervised clustering. Our method
can be applied flexibly to any subset of layers of a DNN architecture and can
incorporate low-level and high-level information. On image datasets given
pre-trained DNNs, we demonstrate the strength of our method in finding similar
training samples, and shedding light on the concepts the DNNs base their
decisions on. Via user studies, we show that our model can improve the user
trust in model's prediction.
