  Explainable AI (XAI) is widely viewed as a sine qua non for ever-expanding AI
research. A better understanding of the needs of XAI users, as well as
human-centered evaluations of explainable models are both a necessity and a
challenge. In this paper, we explore how HCI and AI researchers conduct user
studies in XAI applications based on a systematic literature review. After
identifying and thoroughly analyzing 97core papers with human-based XAI
evaluations over the past five years, we categorize them along the measured
characteristics of explanatory methods, namely trust, understanding, usability,
and human-AI collaboration performance. Our research shows that XAI is
spreading more rapidly in certain application domains, such as recommender
systems than in others, but that user evaluations are still rather sparse and
incorporate hardly any insights from cognitive or social sciences. Based on a
comprehensive discussion of best practices, i.e., common models, design
choices, and measures in user studies, we propose practical guidelines on
designing and conducting user studies for XAI researchers and practitioners.
Lastly, this survey also highlights several open research directions,
particularly linking psychological science and human-centered XAI.
