  Training reliable deep learning models which avoid making overconfident but
incorrect predictions is a longstanding challenge. This challenge is further
exacerbated when learning has to be differentially private: protection provided
to sensitive data comes at the price of injecting additional randomness into
the learning process. In this work, we conduct a thorough empirical
investigation of selective classifiers -- that can abstain when they are unsure
-- under a differential privacy constraint. We find that several popular
selective prediction approaches are ineffective in a differentially private
setting as they increase the risk of privacy leakage. At the same time, we
identify that a recent approach that only uses checkpoints produced by an
off-the-shelf private learning algorithm stands out as particularly suitable
under DP. Further, we show that differential privacy does not just harm utility
but also degrades selective classification performance. To analyze this effect
across privacy levels, we propose a novel evaluation mechanism which isolate
selective prediction performance across model utility levels. Our experimental
results show that recovering the performance level attainable by non-private
models is possible but comes at a considerable coverage cost as the privacy
budget decreases.
