  Very few eXplainable AI (XAI) studies consider how users understanding of
explanations might change depending on whether they know more or less about the
to be explained domain (i.e., whether they differ in their expertise). Yet,
expertise is a critical facet of most high stakes, human decision making (e.g.,
understanding how a trainee doctor differs from an experienced consultant).
Accordingly, this paper reports a novel, user study (N=96) on how peoples
expertise in a domain affects their understanding of post-hoc explanations by
example for a deep-learning, black box classifier. The results show that
peoples understanding of explanations for correct and incorrect classifications
changes dramatically, on several dimensions (e.g., response times, perceptions
of correctness and helpfulness), when the image-based domain considered is
familiar (i.e., MNIST) as opposed to unfamiliar (i.e., Kannada MNIST). The
wider implications of these new findings for XAI strategies are discussed.
