  Large language models (LLMs) have recently shown great advances in a variety
of tasks, including natural language understanding and generation. However,
their use in high-stakes decision-making scenarios is still limited due to the
potential for errors. Selective prediction is a technique that can be used to
improve the reliability of the LLMs by allowing them to abstain from making
predictions when they are unsure of the answer. In this work, we propose a
novel framework for adaptation with self-evaluation to improve the selective
prediction performance of LLMs. Our framework is based on the idea of using
parameter-efficient tuning to adapt the LLM to the specific task at hand while
improving its ability to perform self-evaluation. We evaluate our method on a
variety of question-answering (QA) datasets and show that it outperforms
state-of-the-art selective prediction methods. For example, on the CoQA
benchmark, our method improves the AUACC from 91.23% to 92.63% and improves the
AUROC from 74.61% to 80.25%.
