  Modern Language Models (LMs) are capable of following long and complex
instructions that enable a large and diverse set of user requests. While
Information Retrieval (IR) models use these LMs as the backbone of their
architectures, virtually none of them allow users to provide detailed
instructions alongside queries, thus limiting their ability to satisfy complex
information needs. In this work, we study the use of instructions in IR
systems. First, we introduce our dataset FollowIR, which contains a rigorous
instruction evaluation benchmark as well as a training set for helping IR
models learn to better follow real-world instructions. FollowIR repurposes
detailed instructions -- also known as narratives -- developed for professional
assessors to evaluate retrieval systems. In particular, we build our benchmark
from three collections curated for shared tasks at the Text REtrieval
Conference (TREC). These collections contains hundreds to thousands of labeled
documents per query, making them suitable for our exploration. Through this
process, we can measure how well IR models follow instructions, through a new
pairwise evaluation framework. Our results indicate that existing retrieval
models fail to correctly use instructions, using them for basic keywords and
struggling to understand long-form information. However, we show that it is
possible for IR models to learn to follow complex instructions: our new
FollowIR-7B model has significant improvements after fine-tuning on our
training set.
