  Background: Due to the finite size of the development sample, predicted
probabilities from a risk prediction model are inevitably uncertain. We apply
Value of Information methodology to evaluate the decision-theoretic
implications of prediction uncertainty.
  Methods: Adopting a Bayesian perspective, we extend the definition of the
Expected Value of Perfect Information (EVPI) from decision analysis to net
benefit calculations in risk prediction. In the context of model development,
EVPI is the expected gain in net benefit by using the correct predictions as
opposed to predictions from a proposed model. We suggest bootstrap methods for
sampling from the posterior distribution of predictions for EVPI calculation
using Monte Carlo simulations. In a case study, we used subsets of data of
various sizes from a clinical trial for predicting mortality after myocardial
infarction to show how EVPI changes with sample size.
  Results: With a sample size of 1,000 and at the pre-specified threshold of 2%
on predicted risks, the gain in net benefit by using the proposed and the
correct models were 0.0006 and 0.0011, respectively, resulting in an EVPI of
0.0005 and a relative EVPI of 87%. EVPI was zero only at unrealistically high
thresholds (>85%). As expected, EVPI declined with larger samples. We summarize
an algorithm for incorporating EVPI calculations into the commonly used
bootstrap method for optimism correction.
  Conclusion: Value of Information methods can be applied to explore
decision-theoretic consequences of uncertainty in risk prediction and can
complement inferential methods when developing risk prediction models. R code
for implementing this method is provided.
