  We study the problem of classification with a reject option for a fixed
predictor, applicable in natural language processing. We introduce a new
problem formulation for this scenario, and an algorithm minimizing a new
surrogate loss function. We provide a complete theoretical analysis of the
surrogate loss function with a strong $H$-consistency guarantee. For
evaluation, we choose the decontextualization task, and provide a
manually-labelled dataset of $2\mathord,000$ examples. Our algorithm
significantly outperforms the baselines considered, with a $\sim\!\!25\%$
improvement in coverage when halving the error rate, which is only $\sim\!\! 3
\%$ away from the theoretical limit.
