  In this work, we perform an in-depth analysis of the visualisation methods
implemented in two popular self-explaining models for visual classification
based on prototypes - ProtoPNet and ProtoTree. Using two fine-grained datasets
(CUB-200-2011 and Stanford Cars), we first show that such methods do not
correctly identify the regions of interest inside of the images, and therefore
do not reflect the model behaviour. Secondly, using a deletion metric, we
demonstrate quantitatively that saliency methods such as Smoothgrads or PRP
provide more faithful image patches. We also propose a new relevance metric
based on the segmentation of the object provided in some datasets (e.g.
CUB-200-2011) and show that the imprecise patch visualisations generated by
ProtoPNet and ProtoTree can create a false sense of bias that can be mitigated
by the use of more faithful methods. Finally, we discuss the implications of
our findings for other prototype-based models sharing the same visualisation
method.
