  Recent work has shown that small distilled language models are strong
competitors to models that are orders of magnitude larger and slower in a wide
range of information retrieval tasks. This has made distilled and dense models,
due to latency constraints, the go-to choice for deployment in real-world
retrieval applications. In this work, we question this practice by showing that
the number of parameters and early query-document interaction play a
significant role in the generalization ability of retrieval models. Our
experiments show that increasing model size results in marginal gains on
in-domain test sets, but much larger gains in new domains never seen during
fine-tuning. Furthermore, we show that rerankers largely outperform dense ones
of similar size in several tasks. Our largest reranker reaches the state of the
art in 12 of the 18 datasets of the Benchmark-IR (BEIR) and surpasses the
previous state of the art by 3 average points. Finally, we confirm that
in-domain effectiveness is not a good indicator of zero-shot effectiveness.
Code is available at
https://github.com/guilhermemr04/scaling-zero-shot-retrieval.git
