  This study explores the impact of class outliers on the effectiveness of
example-based explainability methods for black-box machine learning models. We
reformulate existing explainability evaluation metrics, such as correctness and
relevance, specifically for example-based methods, and introduce a new metric,
distinguishability. Using these metrics, we highlight the shortcomings of
current example-based explainability methods, including those who attempt to
suppress class outliers. We conduct experiments on two datasets, a text
classification dataset and an image classification dataset, and evaluate the
performance of four state-of-the-art explainability methods. Our findings
underscore the need for robust techniques to tackle the challenges posed by
class outliers.
