  Background: Risk prediction models are useful tools in clinical
decision-making which help with risk stratification and resource allocations
and may lead to a better health care for patients. AutoScore is a machine
learning-based automatic clinical score generator for binary outcomes. This
study aims to expand the AutoScore framework to provide a tool for
interpretable risk prediction for ordinal outcomes. Methods: The
AutoScore-Ordinal framework is generated using the same 6 modules of the
original AutoScore algorithm including variable ranking, variable
transformation, score derivation (from proportional odds models), model
selection, score fine-tuning, and model evaluation. To illustrate the
AutoScore-Ordinal performance, the method was conducted on electronic health
records data from the emergency department at Singapore General Hospital over
2008 to 2017. The model was trained on 70% of the data, validated on 10% and
tested on the remaining 20%. Results: This study included 445,989 inpatient
cases, where the distribution of the ordinal outcome was 80.7% alive without
30-day readmission, 12.5% alive with 30-day readmission, and 6.8% died
inpatient or by day 30 post discharge. Two point-based risk prediction models
were developed using two sets of 8 predictor variables identified by the
flexible variable selection procedure. The two models indicated reasonably good
performance measured by mean area under the receiver operating characteristic
curve (0.785 and 0.793) and generalized c-index (0.737 and 0.760), which were
comparable to alternative models. Conclusion: AutoScore-Ordinal provides an
automated and easy-to-use framework for development and validation of risk
prediction models for ordinal outcomes, which can systematically identify
potential predictors from high-dimensional data.
