  Copious amounts of relevance judgments are necessary for the effective
training and accurate evaluation of retrieval systems. Conventionally, these
judgments are made by human assessors, rendering this process expensive and
laborious. A recent study by Thomas et al. from Microsoft Bing suggested that
large language models (LLMs) can accurately perform the relevance assessment
task and provide human-quality judgments, but unfortunately their study did not
yield any reusable software artifacts. Our work presents UMBRELA (a recursive
acronym that stands for UMbrela is the Bing RELevance Assessor), an open-source
toolkit that reproduces the results of Thomas et al. using OpenAI's GPT-4o
model and adds more nuance to the original paper. Across Deep Learning Tracks
from TREC 2019 to 2023, we find that LLM-derived relevance judgments correlate
highly with rankings generated by effective multi-stage retrieval systems. Our
toolkit is designed to be easily extensible and can be integrated into existing
multi-stage retrieval and evaluation pipelines, offering researchers a valuable
resource for studying retrieval evaluation methodologies. UMBRELA will be used
in the TREC 2024 RAG Track to aid in relevance assessments, and we envision our
toolkit becoming a foundation for further innovation in the field. UMBRELA is
available at https://github.com/castorini/umbrela.
