  A surge of interest in explainable AI (XAI) has led to a vast collection of
algorithmic work on the topic. While many recognize the necessity to
incorporate explainability features in AI systems, how to address real-world
user needs for understanding AI remains an open question. By interviewing 20 UX
and design practitioners working on various AI products, we seek to identify
gaps between the current XAI algorithmic work and practices to create
explainable AI products. To do so, we develop an algorithm-informed XAI
question bank in which user needs for explainability are represented as
prototypical questions users might ask about the AI, and use it as a study
probe. Our work contributes insights into the design space of XAI, informs
efforts to support design practices in this space, and identifies opportunities
for future XAI work. We also provide an extended XAI question bank and discuss
how it can be used for creating user-centered XAI.
