  Interpreting a large number of neurons in deep learning is difficult. Our
proposed `CLAssifier-DECoder' architecture (ClaDec) facilitates the
understanding of the output of an arbitrary layer of neurons or subsets
thereof. It uses a decoder that transforms the incomprehensible representation
of the given neurons to a representation that is more similar to the domain a
human is familiar with. In an image recognition problem, one can recognize what
information (or concepts) a layer maintains by contrasting reconstructed images
of ClaDec with those of a conventional auto-encoder(AE) serving as reference.
An extension of ClaDec allows trading comprehensibility and fidelity. We
evaluate our approach for image classification using convolutional neural
networks. We show that reconstructed visualizations using encodings from a
classifier capture more relevant classification information than conventional
AEs. This holds although AEs contain more information on the original input.
Our user study highlights that even non-experts can identify a diverse set of
concepts contained in images that are relevant (or irrelevant) for the
classifier. We also compare against saliency based methods that focus on pixel
relevance rather than concepts. We show that ClaDec tends to highlight more
relevant input areas to classification though outcomes depend on classifier
architecture. Code is at \url{https://github.com/JohnTailor/ClaDec}
