  Most popular benchmarks for comparing LLMs rely on a limited set of prompt
templates, which may not fully capture the LLMs' abilities and can affect the
reproducibility of results on leaderboards. Many recent works empirically
verify prompt sensitivity and advocate for changes in LLM evaluation. In this
paper, we consider the problem of estimating the performance distribution
across many prompt variants instead of finding a single prompt to evaluate
with. We introduce PromptEval, a method for estimating performance across a
large set of prompts borrowing strength across prompts and examples to produce
accurate estimates under practical evaluation budgets. The resulting
distribution can be used to obtain performance quantiles to construct various
robust performance metrics (e.g., top 95% quantile or median). We prove that
PromptEval consistently estimates the performance distribution and demonstrate
its efficacy empirically on three prominent LLM benchmarks: MMLU, BIG-bench
Hard, and LMentry. For example, PromptEval can accurately estimate performance
quantiles across 100 prompt templates on MMLU with a budget equivalent to two
single-prompt evaluations. Our code and data can be found at
https://github.com/felipemaiapolo/prompt-eval.
