  Interpreting the inner workings of deep learning models is crucial for
establishing trust and ensuring model safety. Concept-based explanations have
emerged as a superior approach that is more interpretable than feature
attribution estimates such as pixel saliency. However, defining the concepts
for the interpretability analysis biases the explanations by the user's
expectations on the concepts. To address this, we propose a novel post-hoc
unsupervised method that automatically uncovers the concepts learned by deep
models during training. By decomposing the latent space of a layer in singular
vectors and refining them by unsupervised clustering, we uncover concept
vectors aligned with directions of high variance that are relevant to the model
prediction, and that point to semantically distinct concepts. Our extensive
experiments reveal that the majority of our concepts are readily understandable
to humans, exhibit coherency, and bear relevance to the task at hand. Moreover,
we showcase the practical utility of our method in dataset exploration, where
our concept vectors successfully identify outlier training samples affected by
various confounding factors. This novel exploration technique has remarkable
versatility to data types and model architectures and it will facilitate the
identification of biases and the discovery of sources of error within training
data.
