  Recent studies show that large language models (LLMs) can be instructed to
effectively perform zero-shot passage re-ranking, in which the results of a
first stage retrieval method, such as BM25, are rated and reordered to improve
relevance. In this work, we improve LLM-based re-ranking by algorithmically
selecting few-shot demonstrations to include in the prompt. Our analysis
investigates the conditions where demonstrations are most helpful, and shows
that adding even one demonstration is significantly beneficial. We propose a
novel demonstration selection strategy based on difficulty rather than the
commonly used semantic similarity. Furthermore, we find that demonstrations
helpful for ranking are also effective at question generation. We hope our work
will spur more principled research into question generation and passage
ranking.
