  Human evaluation is increasingly critical for assessing large language
models, capturing linguistic nuances, and reflecting user preferences more
accurately than traditional automated metrics. However, the resource-intensive
nature of this type of annotation process poses significant challenges. The key
question driving our work: "is it feasible to minimize human-in-the-loop
feedback by prioritizing data instances which most effectively distinguish
between models?" We evaluate several metric-based methods and find that these
metrics enhance the efficiency of human evaluations by minimizing the number of
required annotations, thus saving time and cost, while ensuring a robust
performance evaluation. We show that our method is effective across widely used
model families, reducing instances of indecisive (or "tie") outcomes by up to
54% compared to a random sample when focusing on the top-20 percentile of
prioritized instances. This potential reduction in required human effort
positions our approach as a valuable strategy in future large language model
evaluations.
