  Background: Effective allocation of limited donor lungs in cystic fibrosis
(CF) requires accurate survival predictions, so that high-risk patients may be
prioritized for transplantation. In practice, decisions about allocation are
made dynamically, using routinely updated assessments. We present a novel tool
for evaluating risk prediction models that, unlike traditional methods,
captures the dynamic nature of decision-making. Methods: Predicted risk is used
as a score to rank incident deaths versus patients who survive, with the goal
of ranking the deaths higher. The mean rank across deaths at a given time
measures time-specific predictive accuracy; when assessed over time, it
reflects time-varying accuracy. Results: Applying this approach to CF Registry
data on patients followed from 1993-2011, we show that traditional methods do
not capture the performance of models used dynamically in the clinical setting.
Previously proposed multivariate risk scores perform no better than forced
expiratory volume in 1 second as a percentage of predicted normal (FEV1%)
alone. Despite its value for survival prediction, FEV1% has a low sensitivity
of 45% over time (for fixed specificity of 95%), leaving room for improvement
in prediction. Finally, prediction accuracy with annually-updated FEV1% shows
minor differences compared to FEV1% updated every 2 years, which may have
clinical implications regarding the optimal frequency of updating clinical
information. Conclusions: It is imperative to continue to develop models that
accurately predict survival in CF. Our proposed approach can serve as the basis
for evaluating the predictive ability of these models by better accounting for
their dynamic clinical use.
