  Literature search questions, such as "where can I find research on the
evaluation of consistency in generated summaries?" pose significant challenges
for modern search engines and retrieval systems. These questions often require
a deep understanding of research concepts and the ability to reason over entire
articles. In this work, we introduce LitSearch, a retrieval benchmark
comprising 597 realistic literature search queries about recent ML and NLP
papers. LitSearch is constructed using a combination of (1) questions generated
by GPT-4 based on paragraphs containing inline citations from research papers
and (2) questions about recently published papers, manually written by their
authors. All LitSearch questions were manually examined or edited by experts to
ensure high quality. We extensively benchmark state-of-the-art retrieval models
and also evaluate two LLM-based reranking pipelines. We find a significant
performance gap between BM25 and state-of-the-art dense retrievers, with a
24.8% difference in absolute recall@5. The LLM-based reranking strategies
further improve the best-performing dense retriever by 4.4%. Additionally,
commercial search engines and research tools like Google Search perform poorly
on LitSearch, lagging behind the best dense retriever by 32 points. Taken
together, these results show that LitSearch is an informative new testbed for
retrieval systems while catering to a real-world use case.
