  With advanced AI/ML, there has been growing research on explainable AI (XAI)
and studies on how humans interact with AI and XAI for effective human-AI
collaborative decision-making. However, we still have a lack of understanding
of how AI systems and XAI should be first presented to users without technical
backgrounds. In this paper, we present the findings of semi-structured
interviews with health professionals (n=12) and students (n=4) majoring in
medicine and health to study how to improve onboarding with AI and XAI. For the
interviews, we built upon human-AI interaction guidelines to create onboarding
materials of an AI system for stroke rehabilitation assessment and AI
explanations and introduce them to the participants. Our findings reveal that
beyond presenting traditional performance metrics on AI, participants desired
benchmark information, the practical benefits of AI, and interaction trials to
better contextualize AI performance, and refine the objectives and performance
of AI. Based on these findings, we highlight directions for improving
onboarding with AI and XAI and human-AI collaborative decision-making.
