  With the increasing deployment of machine learning models in many
socially-sensitive tasks, there is a growing demand for reliable and
trustworthy predictions. One way to accomplish these requirements is to allow a
model to abstain from making a prediction when there is a high risk of making
an error. This requires adding a selection mechanism to the model, which
selects those examples for which the model will provide a prediction. The
selective classification framework aims to design a mechanism that balances the
fraction of rejected predictions (i.e., the proportion of examples for which
the model does not make a prediction) versus the improvement in predictive
performance on the selected predictions. Multiple selective classification
frameworks exist, most of which rely on deep neural network architectures.
However, the empirical evaluation of the existing approaches is still limited
to partial comparisons among methods and settings, providing practitioners with
little insight into their relative merits. We fill this gap by benchmarking 18
baselines on a diverse set of 44 datasets that includes both image and tabular
data. Moreover, there is a mix of binary and multiclass tasks. We evaluate
these approaches using several criteria, including selective error rate,
empirical coverage, distribution of rejected instance's classes, and
performance on out-of-distribution instances. The results indicate that there
is not a single clear winner among the surveyed baselines, and the best method
depends on the users' objectives.
