  Explaining the black-box predictions of NLP models naturally and accurately
is an important open problem in natural language generation. These free-text
explanations are expected to contain sufficient and carefully-selected evidence
to form supportive arguments for predictions. Due to the superior generative
capacity of large pretrained language models, recent work built on prompt
engineering enables explanation generation without specific training. However,
explanation generated through single-pass prompting often lacks sufficiency and
conciseness. To address this problem, we develop an information bottleneck
method EIB to produce refined explanations that are sufficient and concise. Our
approach regenerates the free-text explanation by polishing the single-pass
output from the pretrained language model but retaining the information that
supports the contents being explained. Experiments on two out-of-domain tasks
verify the effectiveness of EIB through automatic evaluation and
thoroughly-conducted human evaluation.
