  Interest in understanding and factorizing learned embedding spaces through
conceptual explanations is steadily growing. When no human concept labels are
available, concept discovery methods search trained embedding spaces for
interpretable concepts like object shape or color that can provide post-hoc
explanations for decisions. Unlike previous work, we argue that concept
discovery should be identifiable, meaning that a number of known concepts can
be provably recovered to guarantee reliability of the explanations. As a
starting point, we explicitly make the connection between concept discovery and
classical methods like Principal Component Analysis and Independent Component
Analysis by showing that they can recover independent concepts under
non-Gaussian distributions. For dependent concepts, we propose two novel
approaches that exploit functional compositionality properties of
image-generating processes. Our provably identifiable concept discovery methods
substantially outperform competitors on a battery of experiments including
hundreds of trained models and dependent concepts, where they exhibit up to 29
% better alignment with the ground truth. Our results highlight the strict
conditions under which reliable concept discovery without human labels can be
guaranteed and provide a formal foundation for the domain. Our code is
available online.
