  Given the wide adoption of ranked retrieval techniques in various information
systems that significantly impact our daily lives, there is an increasing need
to assess and address the uncertainty inherent in their predictions. This paper
introduces a novel method using the conformal risk control framework to
quantitatively measure and manage risks in the context of ranked retrieval
problems. Our research focuses on a typical two-stage ranked retrieval problem,
where the retrieval stage generates candidates for subsequent ranking. By
carefully formulating the conformal risk for each stage, we have developed
algorithms to effectively control these risks within their specified bounds.
The efficacy of our proposed methods has been demonstrated through
comprehensive experiments on three large-scale public datasets for ranked
retrieval tasks, including the MSLR-WEB dataset, the Yahoo LTRC dataset and the
MS MARCO dataset.
