  The performance of large language models (LLMs) is acutely sensitive to the
phrasing of prompts, which raises significant concerns about their reliability
in real-world scenarios. Existing studies often divide prompts into task-level
instructions and case-level inputs and primarily focus on evaluating and
improving robustness against variations in tasks-level instructions. However,
this setup fails to fully address the diversity of real-world user queries and
assumes the existence of task-specific datasets. To address these limitations,
we introduce RobustAlpacaEval, a new benchmark that consists of semantically
equivalent case-level queries and emphasizes the importance of using the worst
prompt performance to gauge the lower bound of model performance. Extensive
experiments on RobustAlpacaEval with ChatGPT and six open-source LLMs from the
Llama, Mistral, and Gemma families uncover substantial variability in model
performance; for instance, a difference of 45.48% between the worst and best
performance for the Llama-2-70B-chat model, with its worst performance dipping
as low as 9.38%. We further illustrate the difficulty in identifying the worst
prompt from both model-agnostic and model-dependent perspectives, emphasizing
the absence of a shortcut to characterize the worst prompt. We also attempt to
enhance the worst prompt performance using existing prompt engineering and
prompt consistency methods, but find that their impact is limited. These
findings underscore the need to create more resilient LLMs that can maintain
high performance across diverse prompts. Data and code are available at
https://github.com/cbwbuaa/On-the-Worst-Prompt- Performance-of-LLMs.
