  Calibration is a vital aspect of the performance of risk prediction models,
but research in the context of ordinal outcomes is scarce. This study compared
calibration measures for risk models predicting a discrete ordinal outcome, and
investigated the impact of the proportional odds assumption on calibration and
overfitting. We studied the multinomial, cumulative, adjacent category,
continuation ratio, and stereotype logit/logistic models. To assess
calibration, we investigated calibration intercepts and slopes, calibration
plots, and the estimated calibration index. Using large sample simulations, we
studied the performance of models for risk estimation under various conditions,
assuming that the true model has either a multinomial logistic form or a
cumulative logit proportional odds form. Small sample simulations were used to
compare the tendency for overfitting between models. As a case study, we
developed models to diagnose the degree of coronary artery disease (five
categories) in symptomatic patients. When the true model was multinomial
logistic, proportional odds models often yielded poor risk estimates, with
calibration slopes deviating considerably from unity even on large model
development datasets. The stereotype logistic model improved the calibration
slope, but still provided biased risk estimates for individual patients. When
the true model had a cumulative logit proportional odds form, multinomial
logistic regression provided biased risk estimates, although these biases were
modest. Non-proportional odds models require more parameters to be estimated
from the data, and hence suffered more from overfitting. Despite larger sample
size requirements, we generally recommend multinomial logistic regression for
risk prediction modeling of discrete ordinal outcomes.
