  We consider the common setting where one observes probability estimates for a
large number of events, such as default risks for numerous bonds.
Unfortunately, even with unbiased estimates, selecting events corresponding to
the most extreme probabilities can result in systematically underestimating the
true level of uncertainty. We develop an empirical Bayes approach "Excess
Certainty Adjusted Probabilities" (ECAP), using a variant of Tweedie's formula,
which updates probability estimates to correct for selection bias. ECAP is a
flexible non-parametric method, which directly estimates the score function
associated with the probability estimates, so it does not need to make any
restrictive assumptions about the prior on the true probabilities. ECAP also
works well in settings where the probability estimates are biased. We
demonstrate through theoretical results, simulations, and an analysis of two
real world data sets, that ECAP can provide significant improvements over the
original probability estimates.
