  Medical applications of machine learning (ML) have experienced a surge in
popularity in recent years. The intensive care unit (ICU) is a natural habitat
for ML given the abundance of available data from electronic health records.
Models have been proposed to address numerous ICU prediction tasks like the
early detection of complications. While authors frequently report
state-of-the-art performance, it is challenging to verify claims of
superiority. Datasets and code are not always published, and cohort
definitions, preprocessing pipelines, and training setups are difficult to
reproduce. This work introduces Yet Another ICU Benchmark (YAIB), a modular
framework that allows researchers to define reproducible and comparable
clinical ML experiments; we offer an end-to-end solution from cohort definition
to model evaluation. The framework natively supports most open-access ICU
datasets (MIMIC III/IV, eICU, HiRID, AUMCdb) and is easily adaptable to future
ICU datasets. Combined with a transparent preprocessing pipeline and extensible
training code for multiple ML and deep learning models, YAIB enables unified
model development. Our benchmark comes with five predefined established
prediction tasks (mortality, acute kidney injury, sepsis, kidney function, and
length of stay) developed in collaboration with clinicians. Adding further
tasks is straightforward by design. Using YAIB, we demonstrate that the choice
of dataset, cohort definition, and preprocessing have a major impact on the
prediction performance - often more so than model class - indicating an urgent
need for YAIB as a holistic benchmarking tool. We provide our work to the
clinical ML community to accelerate method development and enable real-world
clinical implementations. Software Repository:
https://github.com/rvandewater/YAIB.
