  Large language models (LLMs) have demonstrated significant capability to
generalize across a large number of NLP tasks. For industry applications, it is
imperative to assess the performance of the LLM on unlabeled production data
from time to time to validate for a real-world setting. Human labeling to
assess model error requires considerable expense and time delay. Here we
demonstrate that ensemble disagreement scores work well as a proxy for human
labeling for language models in zero-shot, few-shot, and fine-tuned settings,
per our evaluation on keyphrase extraction (KPE) task. We measure fidelity of
the results by comparing to true error measured from human labeled ground
truth. We contrast with the alternative of using another LLM as a source of
machine labels, or silver labels. Results across various languages and domains
show disagreement scores provide a better estimation of model performance with
mean average error (MAE) as low as 0.4% and on average 13.8% better than using
silver labels.
