  Text summarization, a key natural language generation (NLG) task, is vital in
various domains. However, the high cost of inaccurate summaries in
risk-critical applications, particularly those involving human-in-the-loop
decision-making, raises concerns about the reliability of uncertainty
estimation on text summarization (UE-TS) evaluation methods. This concern stems
from the dependency of uncertainty model metrics on diverse and potentially
conflicting NLG metrics. To address this issue, we introduce a comprehensive
UE-TS benchmark incorporating 31 NLG metrics across four dimensions. The
benchmark evaluates the uncertainty estimation capabilities of two large
language models and one pre-trained language model on three datasets, with
human-annotation analysis incorporated where applicable. We also assess the
performance of 14 common uncertainty estimation methods within this benchmark.
Our findings emphasize the importance of considering multiple uncorrelated NLG
metrics and diverse uncertainty estimation methods to ensure reliable and
efficient evaluation of UE-TS techniques.
