  A crucial aspect of understanding the complex nature of Deep Neural Networks
(DNNs) is the ability to explain learned concepts within their latent
representations. While various methods exist to connect neurons to textual
descriptions of human-understandable concepts, evaluating the quality of these
explanation methods presents a major challenge in the field due to a lack of
unified, general-purpose quantitative evaluation. In this work, we introduce
CoSy (Concept Synthesis) -- a novel, architecture-agnostic framework to
evaluate the quality of textual explanations for latent neurons. Given textual
explanations, our proposed framework leverages a generative model conditioned
on textual input to create data points representing the textual explanation.
Then, the neuron's response to these explanation data points is compared with
the response to control data points, providing a quality estimate of the given
explanation. We ensure the reliability of our proposed framework in a series of
meta-evaluation experiments and demonstrate practical value through insights
from benchmarking various concept-based textual explanation methods for
Computer Vision tasks, showing that tested explanation methods significantly
differ in quality.
