  In this paper, we investigate the phenomena of "selection biases" in Large
Language Models (LLMs), focusing on problems where models are tasked with
choosing the optimal option from an ordered sequence. We delve into biases
related to option order and token usage, which significantly impact LLMs'
decision-making processes. We also quantify the impact of these biases through
an extensive empirical analysis across multiple models and tasks. Furthermore,
we propose mitigation strategies to enhance model performance. Our key
contributions are threefold: 1) Precisely quantifying the influence of option
order and token on LLMs, 2) Developing strategies to mitigate the impact of
token and order sensitivity to enhance robustness, and 3) Offering a detailed
analysis of sensitivity across models and tasks, which informs the creation of
more stable and reliable LLM applications for selection problems.
