  Explainable Artificial Intelligence (XAI) systems need to include an
explanation model to communicate the internal decisions, behaviours and actions
to the interacting humans. Successful explanation involves both cognitive and
social processes. In this paper we focus on the challenge of meaningful
interaction between an explainer and an explainee and investigate the
structural aspects of an interactive explanation to propose an interaction
protocol. We follow a bottom-up approach to derive the model by analysing
transcripts of different explanation dialogue types with 398 explanation
dialogues. We use grounded theory to code and identify key components of an
explanation dialogue. We formalize the model using the agent dialogue framework
(ADF) as a new dialogue type and then evaluate it in a human-agent interaction
study with 101 dialogues from 14 participants. Our results show that the
proposed model can closely follow the explanation dialogues of human-agent
conversations.
