  Uncertainty quantification (UQ) in natural language generation (NLG) tasks
remains an open challenge, exacerbated by the intricate nature of the recent
large language models (LLMs). This study investigates adapting conformal
prediction (CP), which can convert any heuristic measure of uncertainty into
rigorous theoretical guarantees by constructing prediction sets, for black-box
LLMs in open-ended NLG tasks. We propose a sampling-based uncertainty measure
leveraging self-consistency and develop a conformal uncertainty criterion by
integrating the uncertainty condition aligned with correctness into the design
of the CP algorithm. Experimental results indicate that our uncertainty measure
generally surpasses prior state-of-the-art methods. Furthermore, we calibrate
the prediction sets within the model's unfixed answer distribution and achieve
strict control over the correctness coverage rate across 6 LLMs on 4 free-form
NLG datasets, spanning general-purpose and medical domains, while the small
average set size further highlights the efficiency of our method in providing
trustworthy guarantees for practical open-ended NLG applications.
