  We present a comparative study between cross-encoder and LLMs rerankers in
the context of re-ranking effective SPLADE retrievers. We conduct a large
evaluation on TREC Deep Learning datasets and out-of-domain datasets such as
BEIR and LoTTE. In the first set of experiments, we show how cross-encoder
rerankers are hard to distinguish when it comes to re-rerank SPLADE on MS
MARCO. Observations shift in the out-of-domain scenario, where both the type of
model and the number of documents to re-rank have an impact on effectiveness.
Then, we focus on listwise rerankers based on Large Language Models --
especially GPT-4. While GPT-4 demonstrates impressive (zero-shot) performance,
we show that traditional cross-encoders remain very competitive. Overall, our
findings aim to to provide a more nuanced perspective on the recent excitement
surrounding LLM-based re-rankers -- by positioning them as another factor to
consider in balancing effectiveness and efficiency in search systems.
