  Advances in AI technologies have resulted in superior levels of AI-based
model performance. However, this has also led to a greater degree of model
complexity, resulting in 'black box' models. In response to the AI black box
problem, the field of explainable AI (xAI) has emerged with the aim of
providing explanations catered to human understanding, trust, and transparency.
Yet, we still have a limited understanding of how xAI addresses the need for
explainable AI in the context of healthcare. Our research explores the
differing explanation needs amongst stakeholders during the development of an
AI-system for classifying COVID-19 patients for the ICU. We demonstrate that
there is a constellation of stakeholders who have different explanation needs,
not just the 'user'. Further, the findings demonstrate how the need for xAI
emerges through concerns associated with specific stakeholder groups i.e., the
development team, subject matter experts, decision makers, and the audience.
Our findings contribute to the expansion of xAI by highlighting that different
stakeholders have different explanation needs. From a practical perspective,
the study provides insights on how AI systems can be adjusted to support
different stakeholders needs, ensuring better implementation and operation in a
healthcare context.
