  As large language models continue to be widely developed, robust uncertainty
quantification techniques will become crucial for their safe deployment in
high-stakes scenarios. In this work, we explore how conformal prediction can be
used to provide uncertainty quantification in language models for the specific
task of multiple-choice question-answering. We find that the uncertainty
estimates from conformal prediction are tightly correlated with prediction
accuracy. This observation can be useful for downstream applications such as
selective classification and filtering out low-quality predictions. We also
investigate the exchangeability assumption required by conformal prediction to
out-of-subject questions, which may be a more realistic scenario for many
practical applications. Our work contributes towards more trustworthy and
reliable usage of large language models in safety-critical situations, where
robust guarantees of error rate are required.
