  From grading papers to summarizing medical documents, large language models
(LLMs) are evermore used for evaluation of text generated by humans and AI
alike. However, despite their extensive utility, LLMs exhibit distinct failure
modes, necessitating a thorough audit and improvement of their text evaluation
capabilities. Here we introduce ALLURE, a systematic approach to Auditing Large
Language Models Understanding and Reasoning Errors. ALLURE involves comparing
LLM-generated evaluations with annotated data, and iteratively incorporating
instances of significant deviation into the evaluator, which leverages
in-context learning (ICL) to enhance and improve robust evaluation of text by
LLMs. Through this iterative process, we refine the performance of the
evaluator LLM, ultimately reducing reliance on human annotators in the
evaluation process. We anticipate ALLURE to serve diverse applications of LLMs
in various domains related to evaluation of textual data, such as medical
summarization, education, and and productivity.
