  In safety-critical applications of machine learning, it is often desirable
for a model to be conservative, abstaining from making predictions on unknown
inputs which are not well-represented in the training data. However, detecting
unknown examples is challenging, as it is impossible to anticipate all
potential inputs at test time. To address this, prior work (Hendrycks et al.,
2018) minimizes model confidence on an auxiliary outlier dataset carefully
curated to be disjoint from the training distribution. We theoretically analyze
the choice of auxiliary dataset for confidence minimization, revealing two
actionable insights: (1) if the auxiliary set contains unknown examples similar
to those seen at test time, confidence minimization leads to provable detection
of unknown test examples, and (2) if the first condition is satisfied, it is
unnecessary to filter out known examples for out-of-distribution (OOD)
detection. Motivated by these guidelines, we propose the Data-Driven Confidence
Minimization (DCM) framework, which minimizes confidence on an uncertainty
dataset. We apply DCM to two problem settings in which conservative prediction
is paramount -- selective classification and OOD detection -- and provide a
realistic way to gather uncertainty data for each setting. In our experiments,
DCM consistently outperforms existing selective classification approaches on 4
datasets when tested on unseen distributions and outperforms state-of-the-art
OOD detection methods on 12 ID-OOD dataset pairs, reducing FPR (at TPR $95\%$)
by $6.3\%$ and $58.1\%$ on CIFAR-10 and CIFAR-100 compared to Outlier Exposure.
