  We present ConceptEvo, a unified interpretation framework for deep neural
networks (DNNs) that reveals the inception and evolution of learned concepts
during training. Our work addresses a critical gap in DNN interpretation
research, as existing methods primarily focus on post-training interpretation.
ConceptEvo introduces two novel technical contributions: (1) an algorithm that
generates a unified semantic space, enabling side-by-side comparison of
different models during training, and (2) an algorithm that discovers and
quantifies important concept evolutions for class predictions. Through a
large-scale human evaluation and quantitative experiments, we demonstrate that
ConceptEvo successfully identifies concept evolutions across different models,
which are not only comprehensible to humans but also crucial for class
predictions. ConceptEvo is applicable to both modern DNN architectures, such as
ConvNeXt, and classic DNNs, such as VGGs and InceptionV3.
