  As machine learning becomes increasingly prevalent in critical fields such as
healthcare, ensuring the safety and reliability of machine learning systems
becomes paramount. A key component of reliability is the ability to estimate
uncertainty, which enables the identification of areas of high and low
confidence and helps to minimize the risk of error. In this study, we propose a
machine learning pipeline called U-PASS tailored for clinical applications that
incorporates uncertainty estimation at every stage of the process, including
data acquisition, training, and model deployment. The training process is
divided into a supervised pre-training step and a semi-supervised finetuning
step. We apply our uncertainty-guided deep learning pipeline to the challenging
problem of sleep staging and demonstrate that it systematically improves
performance at every stage. By optimizing the training dataset, actively
seeking informative samples, and deferring the most uncertain samples to an
expert, we achieve an expert-level accuracy of 85% on a challenging clinical
dataset of elderly sleep apnea patients, representing a significant improvement
over the baseline accuracy of 75%. U-PASS represents a promising approach to
incorporating uncertainty estimation into machine learning pipelines, thereby
improving their reliability and unlocking their potential in clinical settings.
