  The self-rationalising capabilities of LLMs are appealing because the
generated explanations can give insights into the plausibility of the
predictions. However, how faithful the explanations are to the predictions is
questionable, raising the need to explore the patterns behind them further. To
this end, we propose a hypothesis-driven statistical framework. We use a
Bayesian network to implement a hypothesis about how a task (in our example,
natural language inference) is solved, and its internal states are translated
into natural language with templates. Those explanations are then compared to
LLM-generated free-text explanations using automatic and human evaluations.
This allows us to judge how similar the LLM's and the Bayesian network's
decision processes are. We demonstrate the usage of our framework with an
example hypothesis and two realisations in Bayesian networks. The resulting
models do not exhibit a strong similarity to GPT-3.5. We discuss the
implications of this as well as the framework's potential to approximate LLM
decisions better in future work.
