  Machine learning (ML) models deployed in healthcare systems must face data
drawn from continually evolving environments. However, researchers proposing
such models typically evaluate them in a time-agnostic manner, splitting
datasets according to patients sampled randomly throughout the entire study
time period. This work proposes the Evaluation on Medical Datasets Over Time
(EMDOT) framework, which evaluates the performance of a model class across
time. Inspired by the concept of backtesting, EMDOT simulates possible training
procedures that practitioners might have been able to execute at each point in
time and evaluates the resulting models on all future time points. Evaluating
both linear and more complex models on six distinct medical data sources
(tabular and imaging), we show how depending on the dataset, using all
historical data may be ideal in many cases, whereas using a window of the most
recent data could be advantageous in others. In datasets where models suffer
from sudden degradations in performance, we investigate plausible explanations
for these shocks. We release the EMDOT package to help facilitate further works
in deployment-oriented evaluation over time.
