  Despite the impressive capability of large language models (LLMs), knowing
when to trust their generations remains an open challenge. The recent
literature on uncertainty quantification of natural language generation (NLG)
utilises a conventional natural language inference (NLI) classifier to measure
the semantic dispersion of LLMs responses. These studies employ logits of NLI
classifier for semantic clustering to estimate uncertainty. However, logits
represent the probability of the predicted class and barely contain feature
information for potential clustering. Alternatively, CLIP (Contrastive
Language-Image Pre-training) performs impressively in extracting image-text
pair features and measuring their similarity. To extend its usability, we
propose Contrastive Semantic Similarity, the CLIP-based feature extraction
module to obtain similarity features for measuring uncertainty for text pairs.
We apply this method to selective NLG, which detects and rejects unreliable
generations for better trustworthiness of LLMs. We conduct extensive
experiments with three LLMs on several benchmark question-answering datasets
with comprehensive evaluation metrics. Results show that our proposed method
performs better in estimating reliable responses of LLMs than comparable
baselines. Results show that our proposed method performs better in estimating
reliable responses of LLMs than comparable baselines. The code are available at
\url{https://github.com/AoShuang92/css_uq_llms}.
