  Concept Bottleneck Models (CBMs) first map raw input(s) to a vector of
human-defined concepts, before using this vector to predict a final
classification. We might therefore expect CBMs capable of predicting concepts
based on distinct regions of an input. In doing so, this would support human
interpretation when generating explanations of the model's outputs to visualise
input features corresponding to concepts. The contribution of this paper is
threefold: Firstly, we expand on existing literature by looking at relevance
both from the input to the concept vector, confirming that relevance is
distributed among the input features, and from the concept vector to the final
classification where, for the most part, the final classification is made using
concepts predicted as present. Secondly, we report a quantitative evaluation to
measure the distance between the maximum input feature relevance and the ground
truth location; we perform this with the techniques, Layer-wise Relevance
Propagation (LRP), Integrated Gradients (IG) and a baseline gradient approach,
finding LRP has a lower average distance than IG. Thirdly, we propose using the
proportion of relevance as a measurement for explaining concept importance.
