  Should prediction models always deliver a prediction? In the pursuit of
maximum predictive performance, critical considerations of reliability and
fairness are often overshadowed, particularly when it comes to the role of
uncertainty. Selective regression, also known as the "reject option," allows
models to abstain from predictions in cases of considerable uncertainty.
Initially proposed seven decades ago, approaches to selective regression have
mostly focused on distribution-based proxies for measuring uncertainty,
particularly conditional variance. However, this focus neglects the significant
influence of model-specific biases on a model's performance. In this paper, we
propose a novel approach to selective regression by leveraging conformal
prediction, which provides grounded confidence measures for individual
predictions based on model-specific biases. In addition, we propose a
standardized evaluation framework to allow proper comparison of selective
regression approaches. Via an extensive experimental approach, we demonstrate
how our proposed approach, conformalized selective regression, demonstrates an
advantage over multiple state-of-the-art baselines.
