  Unjudged documents or holes in information retrieval benchmarks are
considered non-relevant in evaluation, yielding no gains in measuring
effectiveness. However, these missing judgments may inadvertently introduce
biases into the evaluation as their prevalence for a retrieval model is heavily
contingent on the pooling process. Thus, filling holes becomes crucial in
ensuring reliable and accurate evaluation. Collecting human judgment for all
documents is cumbersome and impractical. In this paper, we aim at leveraging
large language models (LLMs) to automatically label unjudged documents. Our
goal is to instruct an LLM using detailed instructions to assign fine-grained
relevance judgments to holes. To this end, we systematically simulate scenarios
with varying degrees of holes by randomly dropping relevant documents from the
relevance judgment in TREC DL tracks. Our experiments reveal a strong
correlation between our LLM-based method and ground-truth relevance judgments.
Based on our simulation experiments conducted on three TREC DL datasets, in the
extreme scenario of retaining only 10% of judgments, our method achieves a
Kendall tau correlation of 0.87 and 0.92 on an average for Vicu\~na-7B and
GPT-3.5 Turbo respectively.
