  In this paper we present Large Language Model Assisted Retrieval Model
Ranking (LARMOR), an effective unsupervised approach that leverages LLMs for
selecting which dense retriever to use on a test corpus (target). Dense
retriever selection is crucial for many IR applications that rely on using
dense retrievers trained on public corpora to encode or search a new, private
target corpus. This is because when confronted with domain shift, where the
downstream corpora, domains, or tasks of the target corpus differ from the
domain/task the dense retriever was trained on, its performance often drops.
Furthermore, when the target corpus is unlabeled, e.g., in a zero-shot
scenario, the direct evaluation of the model on the target corpus becomes
unfeasible. Unsupervised selection of the most effective pre-trained dense
retriever becomes then a crucial challenge. Current methods for dense retriever
selection are insufficient in handling scenarios with domain shift. Our
proposed solution leverages LLMs to generate pseudo-relevant queries, labels
and reference lists based on a set of documents sampled from the target corpus.
Dense retrievers are then ranked based on their effectiveness on these
generated pseudo-relevant signals. Notably, our method is the first approach
that relies solely on the target corpus, eliminating the need for both training
corpora and test labels. To evaluate the effectiveness of our method, we
construct a large pool of state-of-the-art dense retrievers. The proposed
approach outperforms existing baselines with respect to both dense retriever
selection and ranking. We make our code and results publicly available at
https://github.com/ielab/larmor/.
