  The goal of Explainable AI (XAI) is to design methods to provide insights
into the reasoning process of black-box models, such as deep neural networks,
in order to explain them to humans. Social science research states that such
explanations should be conversational, similar to human-to-human explanations.
In this work, we show how to incorporate XAI in a conversational agent, using a
standard design for the agent comprising natural language understanding and
generation components. We build upon an XAI question bank, which we extend by
quality-controlled paraphrases, to understand the user's information needs. We
further systematically survey the literature for suitable explanation methods
that provide the information to answer those questions, and present a
comprehensive list of suggestions. Our work is the first step towards truly
natural conversations about machine learning models with an explanation agent.
The comprehensive list of XAI questions and the corresponding explanation
methods may support other researchers in providing the necessary information to
address users' demands. To facilitate future work, we release our source code
and data.
