  Large language models have achieved high performance on various question
answering (QA) benchmarks, but the explainability of their output remains
elusive. Structured explanations, called entailment trees, were recently
suggested as a way to explain and inspect a QA system's answer. In order to
better generate such entailment trees, we propose an architecture called
Iterative Retrieval-Generation Reasoner (IRGR). Our model is able to explain a
given hypothesis by systematically generating a step-by-step explanation from
textual premises. The IRGR model iteratively searches for suitable premises,
constructing a single entailment step at a time. Contrary to previous
approaches, our method combines generation steps and retrieval of premises,
allowing the model to leverage intermediate conclusions, and mitigating the
input size limit of baseline encoder-decoder models. We conduct experiments
using the EntailmentBank dataset, where we outperform existing benchmarks on
premise retrieval and entailment tree generation, with around 300% gain in
overall correctness.
