  Abstaining classifiers have the option to abstain from making predictions on
inputs that they are unsure about. These classifiers are becoming increasingly
popular in high-stakes decision-making problems, as they can withhold uncertain
predictions to improve their reliability and safety. When evaluating black-box
abstaining classifier(s), however, we lack a principled approach that accounts
for what the classifier would have predicted on its abstentions. These missing
predictions matter when they can eventually be utilized, either directly or as
a backup option in a failure mode. In this paper, we introduce a novel approach
and perspective to the problem of evaluating and comparing abstaining
classifiers by treating abstentions as missing data. Our evaluation approach is
centered around defining the counterfactual score of an abstaining classifier,
defined as the expected performance of the classifier had it not been allowed
to abstain. We specify the conditions under which the counterfactual score is
identifiable: if the abstentions are stochastic, and if the evaluation data is
independent of the training data (ensuring that the predictions are missing at
random), then the score is identifiable. Note that, if abstentions are
deterministic, then the score is unidentifiable because the classifier can
perform arbitrarily poorly on its abstentions. Leveraging tools from
observational causal inference, we then develop nonparametric and doubly robust
methods to efficiently estimate this quantity under identification. Our
approach is examined in both simulated and real data experiments.
