  Large Language Models (LLMs) have demonstrated remarkable capabilities in
performing complex tasks. Moreover, recent research has shown that
incorporating human-annotated rationales (e.g., Chain-of-Thought prompting)
during in-context learning can significantly enhance the performance of these
models, particularly on tasks that require reasoning capabilities. However,
incorporating such rationales poses challenges in terms of scalability as this
requires a high degree of human involvement. In this work, we present a novel
framework, Amplifying Model Performance by Leveraging In-Context Learning with
Post Hoc Explanations (AMPLIFY), which addresses the aforementioned challenges
by automating the process of rationale generation. To this end, we leverage
post hoc explanation methods which output attribution scores (explanations)
capturing the influence of each of the input features on model predictions.
More specifically, we construct automated natural language rationales that
embed insights from post hoc explanations to provide corrective signals to
LLMs. Extensive experimentation with real-world datasets demonstrates that our
framework, AMPLIFY, leads to prediction accuracy improvements of about 10-25%
over a wide range of tasks, including those where prior approaches which rely
on human-annotated rationales such as Chain-of-Thought prompting fall short.
Our work makes one of the first attempts at highlighting the potential of post
hoc explanations as valuable tools for enhancing the effectiveness of LLMs.
Furthermore, we conduct additional empirical analyses and ablation studies to
demonstrate the impact of each of the components of AMPLIFY, which, in turn,
leads to critical insights for refining in-context learning.
