  As machine learning becomes increasingly integral to autonomous
decision-making processes involving human interaction, the necessity of
comprehending the model's outputs through conversational means increases. Most
recently, foundation models are being explored for their potential as post hoc
explainers, providing a pathway to elucidate the decision-making mechanisms of
predictive models. In this work, we introduce traceable question-answering,
leveraging an external knowledge repository to inform the responses of Large
Language Models (LLMs) to user queries within a scene understanding task. This
knowledge repository comprises contextual details regarding the model's output,
containing high-level features, feature importance, and alternative
probabilities. We employ subtractive counterfactual reasoning to compute
feature importance, a method that entails analysing output variations resulting
from decomposing semantic features. Furthermore, to maintain a seamless
conversational flow, we integrate four key characteristics - social, causal,
selective, and contrastive - drawn from social science research on human
explanations into a single-shot prompt, guiding the response generation
process. Our evaluation demonstrates that explanations generated by the LLMs
encompassed these elements, indicating its potential to bridge the gap between
complex model outputs and natural language expressions.
