  When building recommendation systems, we seek to output a helpful set of
items to the user. Under the hood, a ranking model predicts which of two
candidate items is better, and we must distill these pairwise comparisons into
the user-facing output. However, a learned ranking model is never perfect, so
taking its predictions at face value gives no guarantee that the user-facing
output is reliable. Building from a pre-trained ranking model, we show how to
return a set of items that is rigorously guaranteed to contain mostly good
items. Our procedure endows any ranking model with rigorous finite-sample
control of the false discovery rate (FDR), regardless of the (unknown) data
distribution. Moreover, our calibration algorithm enables the easy and
principled integration of multiple objectives in recommender systems. As an
example, we show how to optimize for recommendation diversity subject to a
user-specified level of FDR control, circumventing the need to specify ad hoc
weights of a diversity loss against an accuracy loss. Throughout, we focus on
the problem of learning to rank a set of possible recommendations, evaluating
our methods on the Yahoo! Learning to Rank and MSMarco datasets.
