  Intuitively, unfamiliarity should lead to lack of confidence. In reality,
current algorithms often make highly confident yet wrong predictions when faced
with relevant but unfamiliar examples. A classifier we trained to recognize
gender is 12 times more likely to be wrong with a 99% confident prediction if
presented with a subject from a different age group than those seen during
training. In this paper, we compare and evaluate several methods to improve
confidence estimates for unfamiliar and familiar samples. We propose a testing
methodology of splitting unfamiliar and familiar samples by attribute (age,
breed, subcategory) or sampling (similar datasets collected by different people
at different times). We evaluate methods including confidence calibration,
ensembles, distillation, and a Bayesian model and use several metrics to
analyze label, likelihood, and calibration error. While all methods reduce
over-confident errors, the ensemble of calibrated models performs best overall,
and T-scaling performs best among the approaches with fastest inference. Our
code is available at https://github.com/lizhitwo/ConfidenceEstimates .
  $\color{red}{\text{Please see UPDATED ERRATA.}}$
