  Unlike in a clinical trial, where researchers get to determine the least
number of positive and negative samples required, or in a machine learning
study where the size and the class distribution of the validation set is static
and known, in a real-world scenario, there is little control over the size and
distribution of incoming patients. As a result, when measured during different
time periods, evaluation metrics like Area under the Receiver Operating Curve
(AUCROC) and Area Under the Precision-Recall Curve(AUCPR) may not be directly
comparable. Therefore, in this study, for binary classifiers running in a long
time period, we proposed to adjust these performance metrics for sample size
and class distribution, so that a fair comparison can be made between two time
periods. Note that the number of samples and the class distribution, namely the
ratio of positive samples, are two robustness factors which affect the variance
of AUCROC. To better estimate the mean of performance metrics and understand
the change of performance over time, we propose a Kalman filter based framework
with extrapolated variance adjusted for the total number of samples and the
number of positive samples during different time periods. The efficacy of this
method is demonstrated first on a synthetic dataset and then retrospectively
applied to a 2-days ahead in-hospital mortality prediction model for COVID-19
patients during 2021 and 2022. Further, we conclude that our prediction model
is not significantly affected by the evolution of the disease, improved
treatments and changes in hospital operational plans.
