  Many ethical frameworks require artificial intelligence (AI) systems to be
explainable. Explainable AI (XAI) models are frequently tested for their
adequacy in user studies. Since different people may have different explanatory
needs, it is important that participant samples in user studies are large
enough to represent the target population to enable generalizations. However,
it is unclear to what extent XAI researchers reflect on and justify their
sample sizes or avoid broad generalizations across people. We analyzed XAI user
studies (n = 220) published between 2012 and 2022. Most studies did not offer
rationales for their sample sizes. Moreover, most papers generalized their
conclusions beyond their target population, and there was no evidence that
broader conclusions in quantitative studies were correlated with larger
samples. These methodological problems can impede evaluations of whether XAI
systems implement the explainability called for in ethical frameworks. We
outline principles for more inclusive XAI user studies.
