  We present the Benchmark of Information Retrieval (IR) tasks with Complex
Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve
documents given multi-faceted user objectives. The benchmark's complexity and
compact size make it suitable for evaluating large language model (LLM)-based
information retrieval systems. We present a modular framework for investigating
factors that may influence LLM performance on retrieval tasks, and identify a
simple baseline model which matches or outperforms existing approaches and more
complex alternatives. No approach achieves satisfactory performance on all
benchmark tasks, suggesting that stronger models and new retrieval protocols
are necessary to address complex user needs.
