  Cross-encoders distilled from large language models (LLMs) are often more
effective re-rankers than cross-encoders fine-tuned on manually labeled data.
However, the distilled models usually do not reach their teacher LLM's
effectiveness. To investigate whether best practices for fine-tuning
cross-encoders on manually labeled data (e.g., hard-negative sampling, deep
sampling, and listwise loss functions) can help to improve LLM ranker
distillation, we construct and release a new distillation dataset:
Rank-DistiLLM. In our experiments, cross-encoders trained on Rank-DistiLLM
reach the effectiveness of LLMs while being orders of magnitude more efficient.
Our code and data is available at
https://github.com/webis-de/msmarco-llm-distillation.
