  Research in explainable AI (XAI) aims to provide insights into the
decision-making process of opaque AI models. To date, most XAI methods offer
one-off and static explanations, which cannot cater to the diverse backgrounds
and understanding levels of users. With this paper, we investigate if free-form
conversations can enhance users' comprehension of static explanations, improve
acceptance and trust in the explanation methods, and facilitate human-AI
collaboration. Participants are presented with static explanations, followed by
a conversation with a human expert regarding the explanations. We measure the
effect of the conversation on participants' ability to choose, from three
machine learning models, the most accurate one based on explanations and their
self-reported comprehension, acceptance, and trust. Empirical results show that
conversations significantly improve comprehension, acceptance, trust, and
collaboration. Our findings highlight the importance of customized model
explanations in the format of free-form conversations and provide insights for
the future design of conversational explanations.
